Suppose you ask an agent to move an appointment to Tuesday afternoon. It replies with a friendly confirmation, the right time, and the right address. But the scheduling system still has your original appointment. Or perhaps the agent moves the appointment correctly, but forgets to tell you the location.

Both interactions need improvement, but the work required is different for each failure mode. One involves the agent's actions, the other involves how it explains their result.

We'll use this hypothetical scheduling agent to work through agent optimization on AWS. Our goal is to get the agent to complete more requests correctly, and to do it with as little work as possible on our side. Amazon Bedrock Advanced Prompt Optimization and Amazon Bedrock AgentCore Optimization are going to help us automate different parts of the process.

This article is sponsored by Inngest

Don't let agent failures blow the budget.

Agents, like all of us, often need to recover from failure. But they shouldn't start from scratch.

Inngest's durable execution checkpoints your agents every step of the way.

So your agents can pick up where they left off, and avoid doubling up on token & compute spend.

Inngest offers a generous free tier and a sleek local dev server for testing.

Choose the behavior you want to improve

You always need to start with a failure mode you can describe precisely. “The agent isn't reliable” doesn't tell you what to change. Instead, “The agent creates a second appointment when asked to reschedule the existing one” gives you something to investigate, and a clear goal to optimize towards.

From there you need to pick examples where that happened, and read the session trace: the record of the model's outputs, tool calls, and returned results. Use that to investigate precisely what went wrong. Did the agent choose the wrong operation? Did it send the wrong appointment ID? Did the correct operation return an error? Each of these causes will require a different fix. AWS's agent debugging guide dives a bit deeper into the topic.

For our scheduling example, the first experiment we need to run depends on what the trace shows:

This is part of what makes agent optimization difficult. Changing one instruction can affect several decisions, and the same agent can behave differently across repeated attempts. You need enough of the execution history to understand where the failure began, so you can know what to do next and how to measure if it's improving. Anthropic's agent evaluation guide makes the useful distinction between the conversation transcript and the final state of the environment. Use both.

Once you know precisely what you want to optimize, you'll know which of these AWS services can help you.

Optimize a model task with Bedrock Advanced Prompt Optimization

Amazon Bedrock Advanced Prompt Optimization, or AdvPO, takes a prompt template, example inputs, and an evaluation method. It runs the examples through your selected models, scores the responses, and uses that feedback to rewrite the template. Then it repeats the process, and keeps improving.

That removes the manual cycle of editing a prompt, running the same examples, comparing responses, and deciding what to try next. You still supply the task and the definition of a good result, it does its best to change the task so its results approach what you define as “good”.

For our agent's incomplete confirmations, AdvPO is a sensible starting point. We can isolate the model call that turns a scheduling result into a message, and focus on that. The input already contains the new appointment details, and the output should communicate them accurately.

Give the optimizer useful examples and feedback

The input file is JSONL, with one record per prompt template. Each record contains its examples and evaluation configuration. Here's an example record, it's expanded for readability but in the actual file it would be a single line. This record contains one prompt template and only two evaluation samples. Add representative samples inside evaluationSamples; add more JSONL lines only when optimizing additional templates.

{
  "version": "bedrock-2026-05-14",
  "templateId": "appointment-confirmation",
  "promptTemplate": "Write a confirmation for this appointment operation.\nRequest: {{request}}\nResult: {{result}}",
  "steeringCriteria": [
    "When rescheduling succeeds, include the appointment day, time, and location.",
    "Do not claim rescheduling succeeded when the result reports failure.",
    "Use only facts provided in the request and result."
  ],
  "evaluationSamples": [
    {
      "inputVariables": [
        {"request": "Move my appointment to Tuesday afternoon."},
        {"result": "status=rescheduled; day=Tuesday; time=14:00; location=Central office"}
      ],
      "referenceResponse": "Your appointment has been moved to Tuesday at 14:00 at the Central office."
    },
    {
      "inputVariables": [
        {"request": "Move my appointment to Tuesday afternoon."},
        {"result": "status=failed; reason=no matching slot available"}
      ],
      "referenceResponse": "I couldn't reschedule your appointment because no matching slot was available."
    }
  ]
}

The {{request}} and {{result}} placeholders receive their values from each sample. Each variable occupies its own object in inputVariables. Reference responses are optional, though highly recommended, since they give the evaluator a concrete example of the desired answer and thus a way to understand what a good answer looks like. You can find the complete schema at the input documentation.

It's important to include examples of failures. If every sample contains a successful reschedule, you give the optimizer little reason to distinguish between confirming an action and explaining that it couldn't happen. Add representative variations, including cases the existing prompt already handles well, and keep some examples aside for checking the result afterward (this is called a held-out set in Machine Learning jargon).

AdvPO provides three evaluation approaches: natural-language steering criteria (which is what the example above uses), a custom LLM judge with your own rubric, or a Lambda function that computes a score. A judge is useful for semantic requirements such as an accurate, complete explanation. Lambda is useful when code can check exact fields, values, or structured output. I wrote about an experiment we ran at work using a Lambda evaluator.

Choose based on what factors make an answer wrong. A beautifully written confirmation containing the wrong appointment time must score poorly, so anything that measures how well-written it is won't work for this case. Remember that the optimizer follows the feedback you give it.

If part of your template already works well, selective optimization lets you mark editable sections with <advpo:optimize>. Content outside those blocks is preserved, and the returned prompt has the tags removed.

Run the job and use the result

Before submitting, make sure your identity has the right permissions to manage optimization jobs, invoke the selected models, and read and write the S3 locations. The S3 bucket where you put the JSONL file must be in the job's AWS Region. Custom Lambda evaluation and KMS encryption add their respective permissions, of course. Here's the list of prerequisites, which also describe cross-Region inference used during optimization.

In the Bedrock console:

1. Open Advanced Prompt Optimization and create a job.

2. Select your target model or models.

3. Upload the JSONL input or provide its S3 location.

4. Choose the S3 output location and submit.

The API equivalent is CreateAdvancedPromptOptimizationJob. If you're trying to improve the prompt, start with your current model. You can also do a model comparison, in that case include the current model and candidate replacements. The service supports up to five models, ten templates per job, and 100 examples per template. Current limits and supported Regions are important to check before preparing a larger input.

The results contain optimized templates for the target models, sample scores, cost estimates, and time to first token. TTFT measures when a streamed response begins. For an agent, also measure how long it takes to provide useful information and complete the requested action. Read the changed responses alongside the scores, then compare the original and selected template on the examples you held aside (your holdout set, in ML jargon).

There are two different costs to consider for AdvPO. The optimization job consumes inference for generating responses, judging them, and rewriting prompts, plus Lambda usage if you choose it. The resulting prompt then has its own recurring inference cost when you use it, which might be higher than the cost of the original prompt if the change that improved the response made your prompt longer. AdvPO has no separate service charge beyond its underlying usage. Check Bedrock pricing for how much models cost.

AdvPO evaluates the supplied examples independently. It can see conversation history or tool results you provide, but it doesn't execute those earlier actions. Our confirmation examples let us test how the model describes a supplied scheduling result. The agent still needs to produce that result correctly, throughout its full execution. That's where AgentCore Optimization comes into play.

Improve agent behavior with AgentCore Optimization

AgentCore Optimization starts with the agent's current configuration and session traces. Its recommendations propose changes to system prompts or tool descriptions, with explanations of the changes.

Let's go back to the case of our agent selecting the create_appointment tool when the user asked to reschedule. If the descriptions make both tools sound like reasonable choices for the action you're asking the agent to take, it's going to get confused about which to pick. A description should make the distinction clear: one creates a new appointment, the other changes an existing appointment identified by its ID. We need to improve that.

Tool-description recommendations analyze selection patterns directly. You provide names, descriptions, and traces, and receive proposed descriptions with explanations. In the AgentCore CLI, use --type tool-description and provide each description with --tools "name:description". This mode omits the evaluator parameter.

System-prompt recommendations work differently. You supply one numerical evaluator to define the direction of improvement, and that's what AgentCore uses to determine whether it's making progress on improving the response. Builtin.GoalSuccessRate is a good fit for task completion, and Builtin.Helpfulness fits more open-ended interaction quality. You can also use a custom evaluator, which can express a requirement specific to your agent. The result is revised prompt text and an explanation. AWS's system-prompt guide documents these choices in more depth.

Generate a candidate from relevant sessions

You can use CloudWatch to get all the necessary data from your agent's past sessions. Configure AgentCore Observability and Transaction Search, invoke your agent, and let the telemetry arrive. You can also supply captured OpenTelemetry-compatible spans from tests, so you don't need a production deployment before you can generate recommendations. See the setup requirements and supported trace sources for more info on how to set this up.

To improve on failures in the agent's overall instructions, the following example illustrates a system-prompt recommendation using seven days of traces. It assumes an existing AgentCore CLI project. Replace MyAgent and the file path with your project's runtime and current prompt.

agentcore run recommendation \
  --type system-prompt \
  --run improve_scheduling \
  --runtime MyAgent \
  --evaluator Builtin.GoalSuccessRate \
  --prompt-file ./system-prompt.txt \
  --lookback 7 \
  --wait

--wait waits for the asynchronous recommendation job. Replace RECOMMENDATION_ID in the following command with the job ID returned above to retrieve the proposed prompt and its explanation:

agentcore view recommendation RECOMMENDATION_ID --json

Per the service quotas, you can send up to 20 sampled sessions per recommendation and a 20,000-character prompt limit. This means that providing a long lookback doesn't mean every session was analyzed. You need to choose evidence that represents the behavior you want to improve, not send every session and hope for the best.

Read the proposed change before applying it (especially in prod). If a recommendation resolves ambiguity by changing what the agent is allowed to do, it hasn't necessarily preserved your intended behavior, even if technically it did improve on the specific issue you're trying to fix. The candidate prompt should still express the task you actually want performed.

Test the changed agent

Apply the candidate to a separate agent version, so you can A/B test and roll back easily if needed. You can deploy another endpoint, or use a configuration bundle: an immutable version of settings such as the prompt, model ID, and tool descriptions. Bundles are optional, but I recommend them. Remember that your agent must read and apply the configuration, just creating one won't replace a prompt hard-coded in your application.

Run the original and candidate agents on the same scenarios, including straightforward rescheduling, ambiguous appointment requests, and unavailable slots. Start each trial with fresh conversation state and independent or reset appointment data and slot availability. Otherwise, the first run can change the task faced by the second.

Check the resulting appointment state and confirmation. Include cases that previously worked so you can make sure that the change that helped one behavior didn't break something else. If the difference isn't clear, repeat the relevant trials and compare outcomes across attempts.

AgentCore batch evaluation scores recorded sessions and returns aggregate results. However, it doesn't take a revised prompt and rerun your agent. Generate fresh sessions with the candidate first, since rescoring the original sessions would only tell you about the original agent. The SDK's dataset runners can handle invocation, telemetry collection, and evaluation together, though that interface is in public preview as of writing this.

Then you should confirm a change under live usage, and for that AgentCore A/B testing splits Gateway traffic between the original version and your new, hopefully improved one. Traffic is sticky by session, keeping every message in a conversation on the same version, so users won't perceive changes in quality within the same session. You can run this for a while and then compare the sessions from both versions.

The experiment results that you can see include per-evaluator changes and statistical uncertainty. Based on this you should check whether the improvement is large enough to matter and whether another relevant measure regressed. With this you'll have enough information to know whether you should deploy the new version or not.

You can also do this for agents hosted outside AgentCore Runtime, but they'll require instrumentation, reachable Gateway targets, and Gateway tracing to associate scored sessions with variants. That's useful when you want managed experiments around an existing agent, but it is setup work to account for.

Recommendations, batch evaluation, and A/B testing are generally available. Recommendation generation has no separate charge, you pay for new evaluations and the underlying resources. If you use batch evaluation, you get a 25% discount from standard evaluation rates. Live experiments consume Gateway, evaluation, and execution resources at their normal rate, with telemetry billed separately. The AgentCore pricing page has the current rates and billing details.

When to use either service, and when to use both

The choice follows the work you need to do:

For example, you might use AdvPO to improve the appointment confirmation in our example agent, and to compare its current model with a less expensive candidate. After selecting a promising template/model combination, you integrate it into the agent. Then you run complete scheduling scenarios with both versions, and use AgentCore's live experiment if you need evidence from real user sessions.

You don't need an AgentCore recommendation job just to validate an AdvPO candidate. Nor do you need AdvPO after AgentCore has already produced a useful recommendation. If your existing test setup can establish the relevant outcomes, AdvPO can be enough for the optimization work. If traces point to unclear tool descriptions, AgentCore can take you from that evidence through a proposed change and its validation.

Conclusion

For a model task you can express through examples and a scoring method, start with Bedrock Advanced Prompt Optimization. For changes informed by agent sessions, use AgentCore recommendations and its evaluation tools. Combine them when you need both prompt/model exploration and managed experiments around the complete agent.

Measure the outputs, and make sure you're not regressing behavior in other scenarios. Also, measure the cost, since it may change significantly. Use A/B testing to test the new version with live data, then promote it if it's better.

You usually use Step Functions when a workflow's complexity outgrows a single Lambda invocation, in terms of duration or of complexity. Lambda durable functions are another path, announced during re:Invent 2025. You write a Lambda handler, and you use a durable execution SDK that checkpoints after each durable operation and replays your code across multiple invocations so the execution can span a much longer duration: from a few minutes to up to one year.

Essentially it's a Lambda programming model + SDK (including JavaScript/TypeScript and Python) that provides a DurableContext with durable operations (steps, waits, callbacks, parallel/map, child contexts). Each durable execution persists a checkpoint log and resumes by replaying from the top and skipping completed operations.

You enable durable execution when creating the Lambda function (you can’t retrofit an existing one), by setting DurableConfig (timeout + retention) via console/CLI/API/IaC; then you deploy code that uses the SDK wrapper/decorator. Then you invoke the function (typically asynchronously) using a qualified version/alias, monitor progress via CloudWatch metrics and EventBridge status-change events, inspect/stop executions via durable execution management commands, and complete callbacks via dedicated Lambda APIs.

Lambda Durable Execution Lifecycle and Replay

A durable execution is a logical run of your workflow that may span many Lambda invocations. The SDK checkpoints after each durable operation, and when the function resumes it replays your handler from the beginning while substituting stored results for already-completed operations. Basically, you write your orchestration code so that anything non-deterministic or side-effecting happens inside durable operations (especially step).

The Four Phases of Lambda Durable Functions

1. Start: A durable execution begins when you invoke the function with durable execution enabled.

2. Checkpoint: Each durable operation (e.g., step, wait, callback) causes a checkpoint record to be persisted before the execution moves on.

3. Suspend and resume: Waits/callbacks intentionally end the current Lambda invocation and schedule/trigger a later resume. Lambda invokes the function again later and the SDK replays to the suspension point.

4. Complete: When your handler returns, the execution completes. Monitoring surfaces succeed/failed/timed out outcomes.

Replay is not a “resume from line N”

Replay means your code runs again from the top, but durable operations short-circuit by returning stored results. Determinism matters a lot in those operations: if you read Date.now(), random IDs, or environment values that change between runs outside a durable operation, you can diverge during replay and hit nondeterminism errors.

Here's a code example using TypeScript:

import { withDurableExecution, DurableContext } from "@aws/durable-execution-sdk-js";

export const handler = withDurableExecution(async (event: any, ctx: DurableContext) => {
  // Pure orchestration logic here (replay-safe).
  const input = event.input;

  const validated = await ctx.step("validate", async () => {
    // Do side effects and nondeterministic reads inside steps.
    return { ok: true, input };
  });

  if (!validated.ok) return { status: "rejected" };

  await ctx.wait("cooldown", { seconds: 30 });

  return { status: "approved" };
});

Now your function is a deterministic program that can be re-run, and durable ops are the boundaries where results are persisted and re-used.

Anyshift, the AI engineer that shows you exactly what broke and why

Production incidents aren't hard because systems are complex. They're hard because nobody has the full picture when things break, and your team ends up burning 45 minutes just rebuilding context that should already exist somewhere.

You could ask AI, but generic copilots don't know your stack. You'll spend more time copypasting logs and filtering through guesses than doing actual root cause analysis.

Unlike other AI SREs, Anyshift automatically maps your cloud, K8s, code, and observability data into a live dependency graph. When something breaks, it already knows how everything connects.

• Root cause in minutes: Solve incidents with full context.

• Noise reduction: P3/P4 alerts are auto-triaged before they hit your phone.

• Proactive safety: Risks are caught before they become incidents.

With security and auditability built into the foundation (SOC 2 Type II certified), it’s a 5-minute setup with no lock-in.

Get started at Anyshift.io

Lambda Durable Operations: steps, waits, callbacks

Durable functions give you three core primitives: steps (checkpointed work with retries), waits (pause without compute billing), and callbacks (pause until an external system responds). You access them via DurableContext methods. Design-wise, you should put all side effects in steps, and use waits/callbacks to “sleep” without tying up a Lambda invocation.

Steps

A step runs code and records its result to the checkpoint log. On replay, completed steps return the stored result instead of re-running. Steps can be retried, which implies they can run more than once unless you configure stricter semantics.

Waits

A wait checkpoints, ends the current invocation, and schedules resumption later. You use waits for backoff, human approval timeouts, polling intervals, etc. The key here is that you stop paying for Lambda compute while waiting.

Callbacks

Callbacks give you a callback ID you can hand to an external system. Your durable execution suspends until that system calls the Lambda callback completion APIs.

The callback ID is the primary identifier in the callback completion APIs (URI path parameter). Treat it as sensitive data and scope who can use it.

Rules of Thumb

• If the code does I/O or side effects, put it in step and make it idempotent.

• If the code needs to wait for some period of time, use wait and don’t poll inside a single Lambda invocation.

• If the code needs to wait for some external call, use callback + completion APIs.

Invocation modes and idempotency keys

Durable functions are designed to be invoked asynchronously, so the caller doesn’t block while the workflow may run for minutes, hours, or days. To prevent duplicate starts, use an execution name as an idempotency key. Within the execution retention period, invoking again with the same name and payload returns information about the existing execution instead of starting another one. Reusing the name with a different payload returns DurableExecutionAlreadyStartedException.

You can supply an execution name when invoking a Lambda Durable function. If an execution with that name is running or completed, an identical payload returns the existing execution information; a different payload produces a DurableExecutionAlreadyStartedException error.

Step Idempotency vs Start Idempotency

Start idempotency prevents duplicate workflow starts. You set it via the execution name. Step idempotency prevents duplicate side effects inside the workflow, and you set it by using step execution mode plus external idempotency keys.

If duplicates come from the event source or caller retries, enforce execution name. That way retries using the same name and payload reuse the existing execution instead of starting a duplicate.

If duplicates come from retries or timeouts within a step, use AT_MOST_ONCE_PER_RETRY and external idempotency tokens. External operations still need idempotency handling; this mode alone does not guarantee exactly-once side effects.

If you can’t define a stable idempotency key, Lambda Durable functions will just end up being very complex, and you should consider Step Functions with explicit task tokens, or a saga pattern for event-driven architectures.

Lambda Durable Functions vs Step Functions

AWS Lambda durable functions are a Lambda execution mode plus a Durable Execution SDK that adds durable primitives (steps, waits, callbacks, etc.) to a single Lambda handler. Your code is replayed from the start on resume, and completed durable operations are skipped using stored checkpoints.

AWS Step Functions is a separate orchestration service where you define a state machine (ASL JSON) and run executions. You wire tasks to AWS services (including Lambda) and get built-in workflow history + visualization.

Execution model and guarantees

In Step Functions, your source of truth is the state machine graph (ASL). Each state transition is managed by Step Functions, which records progress and drives task scheduling. Pricing and limits are centered around state transitions / executions.

Standard workflows guarantee exactly-once workflow execution, and can run up to one year. Asynchronous Express workflows use at-least-once execution; Synchronous Express workflows use at-most-once execution. Both can run for up to five minutes.

For Lambda Durable Functions, your source of truth is the Lambda handler code, but durable operations create checkpoints. On resume, the SDK replays the handler from the beginning and returns stored results for previously completed durable operations. This creates an important constraint: your code must be deterministic across replays (especially any logic outside durable operations). The SDK explicitly warns about replay behavior and determinism.

Waits and human-in-the-loop

Step Functions uses patterns like Wait states or callback (“task token”) patterns. In the callback pattern, Step Functions hands you a taskToken, and an external actor later resumes the workflow by calling APIs like SendTaskSuccess (or the failure variant).

Lambda durable functions, on the other hand, uses durable operations like wait() and callback primitives like createCallback() / waitForCallback(). When waiting, the invocation terminates and later resumes without on-demand compute billed during the wait. External systems resume via Lambda APIs SendDurableExecutionCallbackSuccess / SendDurableExecutionCallbackFailure.

Integrations

Step Functions has native integrations that let you orchestrate many AWS services without creating Lambda functions just to call a service. You also get visual workflow debugging/history (though in my opinion the UI is pretty ugly).

With Lambda Durable Functions you can integrate with anything via AWS SDK calls inside steps (or use context.invoke() to call other Lambda functions as durable operations), which is more natural if you're more used to code. However, you’re building the control flow in code and you won’t get Step Functions’ state-machine-level service integration patterns that would let you just call a service without writing the code for that call.

Cost model

In Step Functions Standard workflows you pay per state transition (retries count too). The free tier includes 4,000 transitions/month (does not expire), and in us-east-1 you pay $0.000025 per transition. Step Functions Express workflows make you pay per request + duration (rounded to 100 ms) + memory used (billed in 64 MB chunks). In us-east-1 you'll pay $1.00 per million requests and $0.00001667 per GB-second (plus tiering at higher GB-hours).

With Lambda Durable Functions you still pay normal Lambda requests + duration (including sub-invocations due to resume/replay), plus durable charges:

• Durable operations: $8.00 per million operations.

• Data written by durable operations (GB): $0.25 per GB.

• Data retained (GB-month, prorated): $0.15 per GB-month.

Decision guidance

Use Step Functions when you need:

• A workflow as a first-class managed artifact (ASL + visualization + execution history).

• Broad AWS-service orchestration with less “glue code” (especially across many services).

• Standard’s semantics and tooling for long-running orchestration (up to 1 year) with exactly-once workflow execution.

Use Lambda durable functions when you need:

• A code-first workflow where the orchestration logic lives in the Lambda handler and you want to use durable primitives (step, wait, waitForCallback, parallel, etc.) with checkpoint+replay.

• To wait without paying for idle compute, and prefer to stay inside the Lambda programming model, and you’re willing to design for determinism/replay.

• Therapy after dealing with Amazon States Language (ASL).

Building a Human Approval Workflow in Step Functions and Lambda Durable Functions

Let's see an example. Say our app needs to process a loan application with the following process:

1. Risk scoring (fast, deterministic-ish compute + external call)

2. If risk is low, approve automatically

3. Else, request human review and pause until the reviewer approves/rejects

4. Finalize decision (write to DB / emit event)

These illustrative examples focus on the “pause + resume via external callback” pattern because it shows the biggest mechanical difference between Step Functions and Lambda Durable Functions. The approval, rejection and timeout paths still need an integrated code review and testing.

Step Functions implementation (Standard workflow + task token callback)

1) State machine definition (ASL JSON)

{
  "Comment": "Loan approval with human-in-the-loop review (Standard)",
  "StartAt": "RiskScore",
  "States": {
    "RiskScore": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": {
        "FunctionName": "${RiskScoreLambdaArn}",
        "Payload.$": "$"
      },
      "OutputPath": "$.Payload",
      "Next": "RiskDecision"
    },
    "RiskDecision": {
      "Type": "Choice",
      "Choices": [
        {
          "Variable": "$.riskScore",
          "NumericLessThanEquals": 30,
          "Next": "AutoApprove"
        }
      ],
      "Default": "RequestHumanApproval"
    },
    "AutoApprove": {
      "Type": "Pass",
      "Parameters": {
        "decision": "APPROVE",
        "reason": "Auto-approved by risk threshold",
        "application.$": "$"
      },
      "Next": "FinalizeDecision"
    },
    "RequestHumanApproval": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke.waitForTaskToken",
      "TimeoutSeconds": 86400,
      "Parameters": {
        "FunctionName": "${RequestApprovalLambdaArn}",
        "Payload": {
          "taskToken.$": "$$.Task.Token",
          "application.$": "$"
        }
      },
      "OutputPath": "$.Payload",
      "Next": "FinalizeDecision"
    },
    "FinalizeDecision": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": {
        "FunctionName": "${FinalizeLambdaArn}",
        "Payload.$": "$"
      },
      "OutputPath": "$.Payload",
      "End": true
    }
  }
}

2) riskScore Lambda

// riskScore.js
export const handler = async (event) => {
  // event: { applicationId, applicant, amount, ... }
  // Keep this deterministic-ish; any nondeterminism should be inside the Lambda call itself (Step Functions won't replay your code).
  const riskScore = Math.min(100, Math.max(0, (event.amount ?? 0) / 1000)); // placeholder logic

  return {
    ...event,
    riskScore,
    scoredAt: new Date().toISOString()
  };
};

3) requestApproval Lambda (sends token to reviewer channel)

This Lambda receives taskToken and must deliver it to your approver UI/system. Step Functions will wait until something calls SendTaskSuccess or SendTaskFailure for that token.

// requestApproval.js
// This function is invoked by Step Functions with a taskToken.
// It should NOT call SendTaskSuccess itself unless you're auto-approving.
// It should send the token to your approval system (email, Slack, ticket, etc.)

export const handler = async (event) => {
  const { taskToken, application } = event;

  // In production: publish to SNS/SQS/EventBridge, create a ticket, notify Slack, etc.
  console.log("Approval requested", {
    applicationId: application.applicationId,
    taskTokenPreview: taskToken.slice(0, 16) + "..."
  });

  // Step Functions "waitForTaskToken" pattern keeps the state open until callback.
  // Return quickly (this Lambda call is just the submission step).
  return {
    status: "PENDING_REVIEW",
    applicationId: application.applicationId
  };
};

4) “Approver callback” Lambda (calls SendTaskSuccess / SendTaskFailure)

Mechanics:

• It receives { taskToken, decision, notes } (e.g., from API Gateway).

• On approve, call SendTaskSuccess with output JSON.

• On reject, call SendTaskFailure.

// approverCallback.js
import { SFNClient, SendTaskSuccessCommand, SendTaskFailureCommand } from "@aws-sdk/client-sfn";

const sfn = new SFNClient({});

export const handler = async (event) => {
  const { taskToken, decision, notes } = event;

  if (!taskToken) throw new Error("Missing taskToken");

  if (decision === "APPROVE") {
    const output = JSON.stringify({ decision: "APPROVE", notes: notes ?? null });
    await sfn.send(new SendTaskSuccessCommand({ taskToken, output }));
    return { ok: true };
  }

  // Reject path: you can send structured error info.
  await sfn.send(
    new SendTaskFailureCommand({
      taskToken,
      error: "RejectedByReviewer",
      cause: notes ?? "No notes"
    })
  );
  return { ok: true };
};

Lambda durable functions implementation (single handler + SDK callback)

0) What you need in code

You wrap your handler with the durable wrapper and use DurableContext operations.

npm install @aws/durable-execution-sdk-js

1) Durable workflow handler (steps + waitForCallback)

This version:

• Calls context.step() for risk scoring and finalization

• Uses context.waitForCallback() to create a callback ID, run the submitter function, and then block until callback completes (or times out)

// durableLoanWorkflow.js
import { withDurableExecution } from "@aws/durable-execution-sdk-js";

/**
 * event: { applicationId, amount, applicant, ... }
 */
export const handler = withDurableExecution(async (event, context) => {
  // Step 1: risk score (checkpointed)
  const scored = await context.step("risk-score", async () => {
    const riskScore = Math.min(100, Math.max(0, (event.amount ?? 0) / 1000)); // placeholder
    return { ...event, riskScore, scoredAt: new Date().toISOString() };
  });

  let decision;
  if (scored.riskScore <= 30) {
    decision = { decision: "APPROVE", notes: "Auto-approved by threshold" };
  } else {
    // Step 2: human approval via callback
    // waitForCallback creates callbackId, runs submitter (send request), then waits.
    decision = await context.waitForCallback(
      "human-approval",
      async (callbackId) => {
        // In production: publish callbackId + context to SNS/SQS/EventBridge, create ticket, etc.
        console.log("Approval requested", {
          applicationId: scored.applicationId,
          callbackIdPreview: callbackId.slice(0, 16) + "..."
        });
      },
      { timeout: { hours: 24 } }
    );
  }

  // Step 3: finalize (checkpointed)
  const finalized = await context.step("finalize-decision", async () => {
    // Write to DynamoDB, emit EventBridge event, etc.
    return {
      applicationId: scored.applicationId,
      riskScore: scored.riskScore,
      ...decision,
      finalizedAt: new Date().toISOString()
    };
  });

  return finalized;
});

Note that this is a single Lambda handler that will be replayed on resume, skipping completed durable operations using checkpoint data. The callback token (“callbackId”) is completed via Lambda callback APIs (SendDurableExecutionCallbackSuccess / Failure).

2) Approver callback function (calls Lambda callback APIs)

Durable callback completion APIs:

• Success: SendDurableExecutionCallbackSuccess takes a CallbackId (URI) and a binary Result payload.

• Failure: SendDurableExecutionCallbackFailure takes a CallbackId and error fields like ErrorType, ErrorMessage, etc.

// durableApproverCallback.js
import {
  LambdaClient,
  SendDurableExecutionCallbackSuccessCommand,
  SendDurableExecutionCallbackFailureCommand
} from "@aws-sdk/client-lambda";

const lambda = new LambdaClient({});

export const handler = async (event) => {
  const { callbackId, decision, notes } = event;
  if (!callbackId) throw new Error("Missing callbackId");

  if (decision === "APPROVE") {
    const resultObj = { decision: "APPROVE", notes: notes ?? null };
    // API expects binary payload; SDK accepts Uint8Array/Buffer for "Result"
    const resultBytes = Buffer.from(JSON.stringify(resultObj), "utf-8");

    await lambda.send(
      new SendDurableExecutionCallbackSuccessCommand({
        CallbackId: callbackId,
        Result: resultBytes
      })
    );

    return { ok: true };
  }

  await lambda.send(
    new SendDurableExecutionCallbackFailureCommand({
      CallbackId: callbackId,
      ErrorType: "RejectedByReviewer",
      ErrorMessage: notes ?? "No notes"
      // ErrorData / StackTrace optional
    })
  );

  return { ok: true };
};

Pricing for Lambda Durable Functions

These values are current as of 2026-02-21, and are for us-east-1. Always check the pricing page.

Lambda durable functions have three pricing components:

1. Lambda compute + requests (same as normal Lambda): You pay for requests and duration, at for example $0.0000133334 per GB-second for ARM, plus $0.20 per million requests. The free tier includes 1M requests/month and 400,000 GB-seconds/month.

2. Durable operations: Each durable operation (execution start, steps, waits, etc.) is metered; the SDK doc shows how operations count by operation type (e.g., Execution Started is 1 op; Step is 1 + retries; WaitForCallback is 3 + retries). You pay $8.00 per million operations.

3. Data written (GB) and data retained (GB-month): Durable operations persist checkpoints; you pay for the data written and for retained storage over time (prorated GB-month). You pay $0.25 per GB written and $0.15 per GB-month of data retained.

Conclusion

Durable functions are a strong fit when you need a long-running workflow but want to stay in Lambda’s programming model.

When to use durable functions

Shipping an LLM-powered feature to production without evals is like deploying software without tests: it works, kind of, hopefully, but not really, and you don't even know how it can fail. This whole article assumes you wouldn't deploy untested code, and it's going to teach you how to apply the same rigor of tests to Generative AI applications.

Traditional code is deterministic: for the same input you get the same output, every time. LLMs aren’t. They sample from a probability distribution over an effectively infinite response space, and don't always produce the same output.

For deterministic tasks that's an implementation challenge, but not really a testing challenge. So long as you know what the expected output should be, you can just test for actual_output == expected_output and easily know whether the LLM says 2+2 equals 4 or 5. The challenge is when you want that variation in the output. When you can't define the expected output, because it's not a single thing, but rather a potentially infinite subset of the larger potentially infinite possible outputs. For example, if you ask AI to write this article. What value would you assign to expected_output in that case?

Btw, no, AI didn't write this article. I did ask, many times. It's still not there yet, not even close. But that's a separate discussion.

So, how do we test that the article the LLM writes is “good”, if we don't have a single value or a finite set of values to compare to? We don't test it, we evaluate it. With evaluations (evals, once you get familiar with them).

Original January 2026 sponsorship (the February 3, 2026 launch event has passed):

Every week, AI makes your team faster—and your data more exposed. Files move to new tools, models train on sensitive inputs, and traditional DLP is blind to the context that matters most.

On February 3 at 11:00 AM PT, we’re pulling back the curtain on Cyberhaven’s unified DSPM & DLP platform—and showing how a single, AI-native platform can finally keep up with how data actually moves.

Join the live launch to see:

• The first public demo of our unified AI & data security platform, built for the realities of 2026 and beyond—SaaS sprawl, shadow AI tools, and data that never sits still.

• How security teams gain x-ray vision into data usage, so they can spot the risky handful of actions hidden in millions of “normal” events—and stop them in real time, not after the damage is done.

• Candid stories from security leaders on where legacy DLP and standalone DSPM break down, and how they’re rethinking data protection around context instead of static rules.

• A look ahead at what’s next for DLP, insider risk, AI security, and DSPM from Cyberhaven’s product and leadership teams—plus where we’re investing next.

Don’t wait for the next AI-related incident to show you the gaps in your data security stack. Reserve your spot and be among the first to see how a unified DSPM + DLP platform can change the way your organization protects its most critical data.

How to Measure Quality in Generative AI Applications

If you ask me to tell you if something is “good”, the first thing I'll do is ask you what do you mean by “good”. We can't define a golden answer (that's what you'd call that expected_output), but we should be able to identify a set of measurable characteristics that we can use to determine if any given answer is “good” or not, in a repeatable way.

Some of these characteristics may be factual accuracy, harmlessness, contextual relevance, or even overall tone. And for each characteristic we need to define how we're going to measure it.

Depending on the metric you're evaluating, you'll have to pick one of these two:

• Reference-based evaluation compares LLM output against predefined ground truth or golden answers. It's great when you can come up with several examples of a good answer, though creating this golden dataset is not as trivial as it sounds, you need to carefully choose examples that represent all the ways in which an answer can be “good” in this specific characteristic.

• Reference-free evaluation evaluates intrinsic qualities without comparing the output to a specific reference. It's a good fit for open-ended questions where the quality is judged on criteria like coherence, relevance, or adherence to guidelines.

And you can pick how you evaluate it (most semantic metrics don't support the first option):

• Automated evaluation uses algorithmic metrics like the presence of certain words or repetition of patterns. These are much faster and cheaper to calculate than more nuanced metrics that rely on semantics, but they're also much less reliable. They're usually a good complement.

• Human evaluation consists of having a human judge the outputs based on certain guidelines. This is a very tried and true method, literally what we've been doing in education, interviews, etc for centuries. It's also slow and expensive, so we generally try to avoid it, but it's useful to know that it's there. Tools like https://lmarena.ai/ use it to generate benchmarks on models.

• Model-based evaluation is like human evaluation, but using an LLM as a judge (hence why we often call it LLM-as-a-Judge). The Judge LLM can understand nuance at a similar level to a human, and we can encode the evaluation guidelines into the prompt. This can be somewhat expensive, to the tune of a few cents or even a few dollars for every run (which is a lot if you compare it to running unit tests), but it's much cheaper than human evaluation.

Let's dive a bit deeper into those.

Automated Evaluation and Programmatic Metrics

Programmatic metrics are scalable, repeatable, very cheap to run, and they catch regressions fast. You'll thank me when you accidentally break formatting in a way that causes everything to crash, and detect it via programmatic metrics. They tend to work reasonably well for Reference-based evaluation.

Foundational lexical metrics

These metrics check for lexical overlap, quantifying the similarity between model-generated text and the references you provide by counting shared word sequences. Basically, they look at the text without really understanding it, at all. They're older than LLMs, but they're still reasonably useful and very cheap.

The most common are BLEU (Bilingual Evaluation Understudy), ROUGE (Recall-Oriented Understudy for Gisting Evaluation) and METEOR (Metric for Evaluation of Translation with Explicit Ordering). There are a few more, but honestly they're boring to explain.

Embedding-based metrics

Lexical metrics fail because different words can mean the same thing. A pretty cool idea to beat that is embedding-based methods, which try to score meaning similarity instead of string similarity.

BERTScore generates contextual embeddings for each token in candidate and reference texts. It computes cosine similarity between each candidate token and the most similar reference token, and then aggregates these scores to calculate embedding-based precision (how well candidate tokens are supported by reference), recall (how well reference tokens are captured by candidate), and F1 score (the harmonic mean of precision and recall).

The advantage of this is that it recognizes paraphrases and synonyms as high-quality matches, instead of looking for literal words. The main limitation is that if there's a one-word difference you'll still get a high score, even if that word is “no” and it changes the meaning of a sentence as a whole.

LLM-as-a-Judge Evaluation

Ideally you'd have reference outputs for everything. However, in some use cases such as very open-ended tasks like creative writing, or when the input space is very large like in multi-turn conversations, a single golden answer can't possibly cover everything that a “good” answer should have. In those situations human evaluation would be the best, if it wasn't so slow and expensive. So we tend to prefer LLM-as-a Judge evaluations.

The idea is to leverage the contextual understanding capabilities of modern LLMs to perform nuanced qualitative assessments. This LLM will evaluate outputs where “good” is subjective and depends on many factors such as helpfulness, coherence, creativity, or adherence to a certain tone.

However, you can't just send a prompt saying “Tell me if this answer is good”. You need your Judge prompt to include:

• The original prompt that produced the output you're evaluating

• The output you're evaluating

• The criteria that output should be evaluated on, including the scale (I recommend binary YES-NO, 1 to 3, or 1 to 5)

• Examples on what each number on the scale means. This is, an output that would be scored 1, an output that would be scored 2, and an output that would be scored 3. You shouldn't use more than 3 to 5 examples, even if you're evaluating multiple criteria

The output of the Judge should include for each criterion:

• The name of the criterion, e.g. clarity, completeness, etc

• The score

• The reasoning

That output should be in JSON, so you can parse it programmatically and calculate metrics like number of evals failed, and use that to fail CI pipelines and create reports just like you do for tests.

Here's an example of a Judge prompt:

Evaluate whether a customer service response provides actionable guidance.

CRITERION

Actionability: Does the response provide clear, specific steps the customer can immediately take?

- YES: Includes concrete actions with sufficient detail to execute
- NO: Remains vague, theoretical, or lacks practical guidance

EVALUATION PROCESS

Think step-by-step:
1. Identify what action the customer needs to take
2. Check if the response provides specific, executable steps
3. Note any vague language ("soon", "should", "might") that reduces actionability
4. Render your verdict with supporting evidence

OUTPUT FORMAT (valid JSON; score must be "YES" or "NO")

{
  "score": "YES",
  "reasoning": "Brief explanation with specific evidence"
}

CALIBRATION EXAMPLES

Example 1 - YES:
- Query: "How do I reset my password?"
- Response: "Click 'Forgot Password' on the login page, enter your email (john@example.com), then check your inbox for a reset link valid for 2 hours."
- Evaluation: `{"score": "YES", "reasoning": "Provides 3 specific steps with concrete details (button name, which email, time limit)"}`

Example 2 - NO:
- Query: "How do I reset my password?"  
- Response: "You can reset your password through our account recovery process. Let me know if you need help!"
- Evaluation: `{"score": "NO", "reasoning": "Says what is possible but provides no steps on how to actually do it"}`

---

NOW EVALUATE

Customer Query: {CUSTOMER_MESSAGE}

Response: {RESPONSE}

Your Evaluation:

You're going to use that Judge prompt to evaluate Actionability. Once you have it, you'll need to collect several relevant CUSTOMER_MESSAGE and RESPONSE pairs, and those are going to be your test cases of sorts. But before you're ready to run these, you need to make sure the Judge agrees with you on your definition of “good”.

Calibrating the Judge

LLM-as-a-Judge evaluation mainly consists of three steps (which you repeat for each evaluation criterion):

1. Figuring out what a “good” answer looks like, to the point where you can look at several answers and reliably answer whether they're good or not

2. Collecting sufficient queries and responses to cover many use cases relevant to your application

3. Creating a Judge prompt that gives to those responses the same score that you would give them (this is calibrating the Judge)

That's why you want that “reasoning” field in the response. You need to make sure the Judge's score agrees with your score. You'll find that the main limitation to this isn't really in how you write the Judge prompt, but rather in your own understanding of why you would pick a certain score, and overall what a “good” answer looks like.

LLM Evaluation Frameworks and Tools

Amazon Bedrock Evaluations is AWS's option for evals. You can use Amazon Bedrock directly to generate the outputs and evaluate them, or you can Bring Your Own Inference, importing a JSONL file with your prompts and outputs, and letting Amazon Bedrock run automated metrics and LLM-as-a-Judge evaluations.

Another good option is LangChain's OpenEvals, especially if you're using LangChain already. Though if you're using Python I prefer DeepEval, which lets you treat evals like unit tests (they're more akin to functional tests in my opinion, but that's besides the point, they're definitely tests). And if you like TypeScript, I've used Promptfoo with a couple of customers and it works very well. There are more tools around though. Treat them like you'd treat a unit tests framework: try a few if you want, pick one, and mainly focus on the tests themselves (or the evals themselves in this case) instead of on the framework.

Evals in the Development Lifecycle

Evals are tests for LLMs. That's the role they play in Generative AI applications. And like tests, the real value isn't in using them once to test one single version, but in running them every single time you make a change, giving you confidence that you're not inadvertently breaking something.

The difference between evals and tests (besides evals being a lot harder to write) is that you shouldn't aim for a perfect score with evals. If you get a perfect score, it's likely that your evals are not testing some edge cases.

Instead of aiming for a perfect score, what you need to do is define a threshold of acceptable quality. You can set for each metric a minimum value that all evals must meet, e.g. a minimum score of 4 in a 1-5 scale, plus a threshold for the average score across all evals in that metric, such as 4.5/5. This gives you a reliable quality gate that lets you catch regressions.

Moreover, it gives you room to improve upon your prompts. You can calculate your evals scores for the current version of a prompt, make some changes, and run the evals suite again. Comparing the score will tell you if the change produces better, worse, or equivalent results. You can use this to improve your evals scores, to deal with specific edge cases without regressing on other use cases, and even to try to reduce your prompt without losing quality.

You should also use evals to evaluate models. Everyone knows Claude Opus 4.5 is “better” than Claude Sonnet 4.5, though it's also more expensive. With a good evals suite you can get a score for both models and determine precisely how much better Opus is for your specific application, and make an informed decision based on price and actual performance for you, not on a generic and probably gamed benchmark. Again, use this to improve the scores, or to find the cheapest model that gives you acceptable scores.

Of course, don't (or don't just) do this in your computer, as an experiment. To be effective at catching regressions, evals should be a quality gate in your CI pipeline, just like normal tests are. Pick a threshold, and commit to never going below that level of quality. If a change pulls you down, it's a regression and it should be considered equivalent to introducing a bug.

Conclusion

The LLM is part of your Generative AI application, and evals are the only tests you can write for it. Moreover, as you write your evals you'll find that defining what “good” means helps you improve your application significantly, at least in my experience.

Treat evals like tests, use them to catch regressions, and commit to a certain level of quality, failing your CI if you go below it. Vibes are fine to get started, but if you want to build serious software, make quality measurable in a repeatable way. And evaluate it. With evals.

DynamoDB is not a relational database, and you can't design for it like one. Hopefully this isn't news to you, it must be the 12th time I say it. But I also understand this isn't super useful: “Don't design like a relational DB” doesn't really tell you what to do.

In this guide, we're going deep into DynamoDB data modeling. We'll cover how to think about your data, how to structure it for the access patterns your application actually needs, and how recent features (like multi-attribute composite keys in GSIs and the updated guidance on single-table vs. multi-table design) can make your life easier. This isn't a "Getting Started with DynamoDB" tutorial, I'll be assuming you already know the basics. We're here to learn how to design your data properly.

Why Database Design Matters More in DynamoDB

With Postgres or MySQL, you can design your schema, normalize it, and then figure out your queries later. Sure, you might need to add an index here and there, but the database will figure out how to execute your query. It might not be fast, but it'll work, and you can almost always optimize later.

DynamoDB doesn't work that way. There's no query optimizer that figures out how to satisfy your request. Either your data is structured to answer a specific question efficiently, or it's not. And if it's not, you're paying for a full table scan, which is the difference between milliseconds and minutes (and between reasonable costs and a terrifying AWS bill).

So here's the fundamental difference: in relational databases, you design your schema first and figure out queries later. In DynamoDB, you design your schema around the queries you need to run. Access patterns come first, schema comes second.

Understanding the Basics of DynamoDB

Before we get into the design process, let's make sure we're on the same page about how DynamoDB stores and retrieves data. (If you already know this, feel free to skip ahead.)

Partition Keys and Sort Keys

Every item in DynamoDB needs a primary key. This can be either:

• Simple primary key: Just a partition key (PK). Each item must have a unique partition key value.

• Composite primary key: A partition key (PK) plus a sort key (SK). The combination must be unique.

The partition key determines which physical partition your data lives on. DynamoDB hashes the partition key and uses that hash to route your request to the right server. This is how DynamoDB scales horizontally: it adds more partitions as your data grows. (You don't see this happening, which is kind of the whole point of a managed service.)

The sort key, when present, determines the order of items within a partition. Items with the same partition key are stored together (this is called an item collection), sorted by the sort key. This co-location is incredibly useful because it lets you retrieve multiple related items with a single query.

Global Secondary Indexes (GSIs) and Local Secondary Indexes (LSIs)

Sometimes you need to query your data in ways that your primary key doesn't support. That's where indexes come in.

Local Secondary Indexes use the same partition key as your table but a different sort key. They must be created when you create the table, you can't add them later, and they share the partition's throughput capacity. LSIs are useful when you need an alternative sort order within the same partition.

Global Secondary Indexes can have a completely different partition key and sort key from the base table, and you can create or delete them at any time. GSIs have their own throughput capacity, separate from the base table. They're more flexible but also more expensive: you're essentially duplicating your data into a new structure (and you're paying for the write operations to do that).

Here's a recent development worth knowing: as of November 2025, DynamoDB supports multi-attribute composite keys in GSIs. Instead of concatenating values into synthetic keys like TOURNAMENT#WINTER2024#REGION#NA-EAST, you can now use up to four attributes each for the partition key and sort key. This makes your data model cleaner and eliminates the need to backfill synthetic keys when adding new indexes. More on this later.

The Access Patterns Approach

Here's how you should approach DynamoDB design:

1. Identify your entities. What are the "things" your application deals with? Customers, orders, products, invoices, etc.

2. Map the relationships. How do these entities relate to each other? One-to-many? Many-to-many?

3. List your access patterns. This is the critical step. What questions does your application need to answer? Be specific. Not "get orders" but "get all orders for a customer in the last 30 days."

4. Design your keys to satisfy those patterns. This is where the actual schema design happens.

5. Add indexes for patterns your primary key can't handle. But only if you actually need them.

If you want to visualize your design before writing code, NoSQL Workbench (free tool created by AWS) lets you model your table and test queries against sample data. I find it invaluable for validating designs before committing to them.

Let's work through a real example. We're building an e-commerce application. Here are our entities:

• Customers: People who buy stuff

• Products: Stuff people buy

• Orders: When a customer buys stuff

• Order Items: The individual products within an order

• Invoices: Payment records for orders

And here are our access patterns:

1. Get customer by customerId

2. Get product by productId

3. Get order by orderId

4. Get all products in an order

5. Get the invoice for an order

6. Get all orders containing a specific product (for a date range)

7. Get an invoice by invoiceId

8. Get all invoices for a customer (for a date range)

9. Get all products ordered by a customer (for a date range)

Now we design our schema to satisfy these patterns.

Single-Table vs. Multi-Table Design

Before we dive into the actual design, we need to talk about a philosophical question in DynamoDB: should you put everything in one table, or use multiple tables?

Single-table design stores multiple entity types in the same table. You use generic attribute names like PK and SK for your keys (yes, you literally name your PK PK, and your SK SK), and an EntityType attribute to distinguish between customers, orders, products, etc. This approach was heavily promoted by AWS for years, because it lets you fetch related entities with a single query.

Multi-table design is more like traditional database design: one table per entity type. It's simpler to understand and implement, especially if you're coming from a relational background.

My recommendation is a hybrid: Single table per domain. Single-table design works well when you frequently need to query multiple entity types together, and multi-table design is fine when your access patterns don't require cross-entity queries. So my recommendation is to group in a single table entities that need to be queried together, and end up with a handful of tables, each of which can resolve the queries of a single domain (meaning a group of entities that exist together).

For our e-commerce example, we're going to use single-table design because several of our access patterns need to fetch related entities together (like getting an order with all its items), and because it's simpler. In a larger application, we'd end up with multiple “single” tables.

Designing the E-Commerce Schema

Let's implement our access patterns one by one. I'll show you how the table evolves as we add support for each pattern.

Access Patterns 1 & 2: Get Customer or Product by ID

These are the simplest patterns. We store customers with PK=c#<customerId> and SK=c#<customerId>. Products get PK=p#<productId> and SK=p#<productId>.

Why the prefixes? Because we're storing multiple entity types in the same table, and we need the keys to be unambiguous. Also, this makes it easy to identify the entity type at a glance.

To get a customer: Query where PK = "c#12345" AND SK = "c#12345"

Access Patterns 3 & 4: Get Order and Its Products

Here's where single-table design starts to shine. An order and its items share the same partition key (o#<orderId>), but they have different sort keys. The order itself has SK=o#<orderId>, while each order item has SK=p#<productId>.

To get everything about an order (the order itself plus all items): Query where PK = "o#001"

To get just the products in an order: Query where PK = "o#001" AND SK begins_with "p#"

This is the power of item collections. All items with the same partition key are stored together, so DynamoDB can retrieve related items with Query, using pagination when results exceed one page. No joins needed.

Here's what this looks like in code:

import boto3
from boto3.dynamodb.conditions import Key

dynamodb = boto3.resource('dynamodb')
table = dynamodb.Table('ECommerce')

# Read the first page of order metadata and items; use LastEvaluatedKey for additional pages
response = table.query(
    KeyConditionExpression=Key('PK').eq('o#001')
)

# Separate order from items based on EntityType
order = None
items = []
for item in response['Items']:
    if item['EntityType'] == 'order':
        order = item
    elif item['EntityType'] == 'orderItem':
        items.append(item)

# Or get just the products in the order
response = table.query(
    KeyConditionExpression=Key('PK').eq('o#001') & Key('SK').begins_with('p#')
)

One query, multiple entity types, no extra per-entity requests for results that fit in one page. That's the payoff for the design complexity.

Still with me? This is where it starts to get interesting.

Access Pattern 5: Get Invoice for an Order

The invoice belongs to the order, so it goes in the same item collection:

To get the invoice for an order: Query where PK = "o#001" AND SK begins_with "i#"

Access Pattern 6: Get Orders for a Product (Date Range)

This is our first pattern that requires a GSI. We need to query by product ID and filter by date, but our base table doesn't support that: products are partition keys for products, not for order items. (We've hit the wall that every DynamoDB designer eventually hits.)

We'll create a GSI (let's call it GSI1) where:

• Partition key: productId

• Sort key: orderDate

Wait—but we're using single-table design with generic key names. How does this work?

We add attributes to our order items: GSI1PK (the product ID) and GSI1SK (the order date). DynamoDB will index only items that have these attributes, creating what's called a sparse index. Items without these attributes won't appear in the GSI, and you don't need to pay to store them twice.

To get all orders containing product p#001 in March 2025: Query GSI1 where GSI1PK = "p#001" AND GSI1SK BETWEEN "2025-03-01" AND "2025-03-31"

Access Patterns 7 & 8: Get Invoice by ID, Get Payments for Invoice

Invoice by ID is interesting. Our invoice items are currently keyed by order ID, but we also need to retrieve them by invoice ID. Time for another entry in our GSI.

We'll add GSI1PK=i#<invoiceId> and GSI1SK=i#<invoiceId> to our invoice items:

For payments, we could create separate payment items, but since our access pattern is "get all payments for an invoice" (not individual payments), we can store payments as a JSON attribute on the invoice. This is a valid denormalization, we're trading flexibility for simplicity.

Access Patterns 8 & 9: Customer Invoice and Product History

These patterns need customer ID as the partition key, which our current structure doesn't support. Time for a second GSI (GSI2):

The sort key includes both a prefix (to distinguish invoices from products) and a date (for range queries).

To get all invoices for customer c#12345 in Q1 2025: Query GSI2 where GSI2PK = "c#12345" AND GSI2SK BETWEEN "i#2025-01-01" AND "i#2025-03-31"

To get all products ordered by customer c#12345 in Q1 2025: Query GSI2 where GSI2PK = "c#12345" AND GSI2SK BETWEEN "p#2025-01-01" AND "p#2025-03-31"

If this feels like a lot of moving parts, that's because it is—but you're almost through the hard part.

Making It Cleaner with Multi-Attribute Keys

Remember when I mentioned multi-attribute composite keys in GSIs? Here's where they can help. (This feature shipped in November 2025, so it's relatively new.)

Without multi-attribute keys, we created synthetic keys like GSI2SK = "i#2025-03-15" by concatenating a prefix with a date. This works, but it has drawbacks:

• You need to build and parse these strings in your application code (error-prone and tedious)

• Adding a new GSI requires backfilling synthetic keys across all existing items

• The data model is harder to understand at a glance

With multi-attribute keys, you can define a GSI where the sort key is composed of multiple actual attributes. For example, instead of GSI2SK = "i#2025-03-15", you could have:

• Sort key attribute 1: entityType (value: "invoice")

• Sort key attribute 2: orderDate (value: "2025-03-15")

The query becomes more natural: Query where GSI2PK = "c#12345" AND entityType = "invoice" AND orderDate BETWEEN "2025-01-01" AND "2025-03-31"

You can use up to four attributes in the partition key and four in the sort key.

The Critical Query Constraint

Here's what you absolutely need to know: multi-attribute sort key queries must be left-to-right. You must specify conditions in order, and you cannot skip attributes in the middle.

Valid query patterns:

• ✅ First attribute only: entityType = 'invoice'

• ✅ First two attributes: entityType = 'invoice' AND orderDate = '2025-03-15'

• ✅ First attribute equality + second attribute range: entityType = 'invoice' AND orderDate BETWEEN '2025-01-01' AND '2025-03-31'

Invalid query patterns:

• ❌ Skipping first attribute: orderDate = '2025-03-15' (can't skip entityType)

• ❌ Range on first, equality on second: entityType > 'invoice' AND orderDate = '2025-03-15' (inequality must be last)

• ❌ Querying out of order: orderDate = '2025-03-15' AND entityType = 'invoice'

This constraint directly affects how you order attributes in your key definition. Put the attributes you'll always filter on first, with the most common range query attribute last.

This feature is particularly useful when:

• You're adding a GSI using attributes already present on existing items (no application backfill of synthetic key attributes required)

• Your data naturally has multiple hierarchical dimensions

• You want your schema to be more self-documenting

One thing to keep in mind: multi-attribute keys only work with GSIs, not with the base table or LSIs. And you still need to think carefully about data distribution: a multi-attribute partition key that results in a few "hot" partitions will cause throttling just like a single-attribute key would.

The Final Schema

Here's what our complete design looks like.

Base Table:

GSI1 (for product-based and invoice-based queries):

GSI2 (for customer-based queries):

Access Pattern Summary:

Design Principles to Remember

After walking through this example, let me distill some principles that apply to any DynamoDB design. These are worth bookmarking.

1. Start with access patterns. Never start by drawing an ERD and then trying to fit it into DynamoDB. List your queries first, then design a schema that supports them. This feels backwards if you're used to relational databases, but it's the right approach for DynamoDB.

2. Use composite keys to create item collections. Items that are often queried together should share a partition key. The sort key distinguishes them and controls their order.

3. Add indexes only when necessary. Each GSI duplicates your data and adds cost. Make sure you actually need the access pattern before creating an index. A surprising number of "necessary" indexes turn out to be premature optimization.

4. Embrace denormalization. Storing the same data in multiple places (like customer ID on both orders and order items) is normal and expected. Storage is cheap, round trips are expensive.

5. Use sparse indexes. If a GSI only needs to index certain item types, only add the GSI key attributes to those items. This keeps the index smaller and cheaper.

6. Plan for growth. Choose partition keys that will distribute load evenly as your data grows. A partition key with low cardinality (like "status" with values "active/inactive") will create hot partitions. You'll regret it later.

7. Consider using multi-attribute keys. Especially for GSIs on existing tables, they can simplify your schema and avoid application backfills of synthetic keys when the required attributes already exist. This is one of those features that, once you use it, you wonder how you lived without it.

8. Remember transactions cost double. DynamoDB transactions (TransactWriteItems, TransactGetItems) consume 2x the capacity units of regular operations. If your access patterns involve frequent transactional writes across multiple items, factor this into your capacity planning and cost estimates.

9. Use TTL for time-bound data. For time-series data, session records, logs, or anything with natural expiration, enable Time to Live. DynamoDB deletes expired items asynchronously without charging source-Region write capacity. On MREC global tables, replicated TTL deletes consume write capacity in replica Regions. You'll need a TTL attribute (typically a Unix timestamp), so plan for it in your schema.

Recent Features That Affect Design Decisions

DynamoDB has evolved significantly in the past year or so. Here are some features worth knowing about, they might change how you approach certain design decisions.

Warm Throughput

DynamoDB now exposes a "warm throughput" value that shows how many read and write operations your table can readily handle. More importantly, you can pre-warm your table to handle anticipated traffic spikes. This is useful if you know a traffic surge is coming (say, a product launch or Black Friday) and want to avoid throttling during the critical window.

Configurable Maximum Throughput for On-Demand Tables

On-demand tables now support configurable maximum throughput. This lets you set a ceiling on how much capacity a table can consume, which is helpful for cost control and for protecting downstream services from being overwhelmed.

Multi-Region Strong Consistency

Global tables now support multi-region strong consistency. Previously, global tables only offered eventual consistency: you'd write in one region, and it would eventually replicate to others. With multi-region strong consistency, you can get strong consistency across regions, which is crucial for applications like inventory management or financial transactions. This is a big deal for applications that previously couldn't use global tables due to consistency requirements. Check the supported MRSC Region combinations and feature restrictions before choosing it; MRSC does not support TTL or transaction operations.

Configurable PITR Periods

Point-in-time recovery (PITR) now lets you configure the recovery period from 1 to 35 days (it used to be fixed at 35 days). This is helpful for compliance scenarios that require shorter data retention. The change came in January 2025, and it's one of those "finally!" features.

Zero-ETL Integrations

DynamoDB now has zero-ETL integrations with Amazon Redshift and SageMaker Lakehouse. Changes to your DynamoDB table are automatically replicated to these analytics services, so you can run complex queries and ML workloads without building ETL pipelines. If you've ever struggled with the "DynamoDB is great for OLTP but how do I run analytics?" question, this is your answer.

Common Questions

Let me anticipate some questions you might have. These come up every time I discuss DynamoDB design.

Why use one table instead of multiple ones?

Because related data stays together. In DynamoDB, there's no join operation: if you need data from multiple tables, you need to make multiple API calls. Single-table design lets you fetch related entities in one query.

That said, multi-table design is perfectly valid when your access patterns don't require cross-entity queries. AWS now explicitly supports both approaches. Don't let anyone tell you it's "wrong."

Is there only one correct design?

No. There are usually multiple valid designs, and the "best" one depends on your specific access patterns, expected data volume, and team familiarity with DynamoDB. A good design satisfies all your access patterns efficiently, but there's rarely only one way to achieve that.

How do I handle new access patterns?

If the new pattern can be supported by your existing keys or indexes, great, you're done. If not, you'll need to add a GSI. With multi-attribute composite keys, you might not need to backfill data, you can use existing attributes as key components.

If your data model fundamentally can't support the new pattern (like needing to filter by an attribute that doesn't exist), you might need to add that attribute and backfill it across existing items. This is annoying but manageable.

Should I use DynamoDB for everything?

No. DynamoDB is excellent for high-scale, low-latency access patterns that you can define in advance. It's not great for ad-hoc analytics, complex reporting, or applications where you don't know your access patterns upfront. If your stakeholders say "we'll figure out the queries later," run towards relational and don't look back.

For analytics on DynamoDB data, consider the zero-ETL integrations with Redshift or exporting to S3 and using Athena.

Is DynamoDB really this complex?

Not really. What's complex is unlearning relational database habits. Once you internalize the "access patterns first" approach, DynamoDB design becomes surprisingly methodical. You're not normalizing and then optimizing, you're designing for your specific queries from the start.

The good news: once your design is done, operating DynamoDB is significantly simpler than managing a relational database cluster. No sharding, no replication configuration, no maintenance windows. Well, none that you manage, AWS handles all of it.

Wrapping Up

DynamoDB database design is fundamentally different from relational database design. Instead of normalizing data and writing queries later, you start with the queries and design your schema to support them.

The key concepts are:

• Partition keys for data distribution and direct lookups

• Sort keys for ordering and range queries within a partition

• Item collections for co-locating related entities

• GSIs for alternative access patterns

• Single-table design for data locality (when your patterns benefit from it)

Recent features like multi-attribute composite keys, configurable throughput, and warm throughput make DynamoDB more flexible and easier to operate. But the core design principles remain the same: know your access patterns, structure your data to serve them, and embrace denormalization.

It's 3 am, your phone buzzes. CloudWatch alarm: DynamoDB throttling. You check the metrics: your table has 10,000 Write Capacity Units (WCUs) provisioned, but you're barely using 3,000. The graphs show plenty of headroom. So why are requests getting rejected?

Hopefully this doesn't happen to you (especially not at 3 am). But if it does, you should know the problem isn't your table's total capacity, but that every DynamoDB table is split into physical partitions, and each partition has a hard ceiling of 1,000 WCUs per second. When one partition key gets hammered, that specific partition throttles, even if your other 99 partitions are sitting idle.

Understanding this partition-level reality is the difference between a system that scales smoothly under load and one that falls apart during your biggest traffic spike of the year. Let's get down to that. Review note: the key-design examples, scaling explanation, counter correctness and cost model below need a coordinated revision; treat the code as illustrative.

The 1,000 WCU Ceiling You Can't See

DynamoDB's architecture is built on physical partitions. These are the fundamental storage and throughput units that actually hold your data. Each partition is subject to three hard limits:

• 1,000 Write Capacity Units per second

• 3,000 Read Capacity Units per second

• 10 GB of storage

These limits are per partition, not per table. When you provision 10,000 WCUs for your table, DynamoDB distributes that capacity across multiple partitions. If your table has 10 physical partitions, each partition gets roughly 1,000 WCUs to work with (the distribution isn't always perfectly even, but that's the general idea).

Your partition key (PK) determines which physical partition stores each item. If your application writes heavily to a small number of partition keys (maybe you're tracking views on a viral post, or incrementing a global counter, or processing a massive batch of orders for a single customer) those writes all target the same partition. Once that partition hits its 1,000 WCU ceiling, requests start getting throttled with ProvisionedThroughputExceededException errors.

If your CloudWatch metrics show the table is only using 30% of its provisioned capacity, that's because the other partitions are fine, they're not receiving traffic. But DynamoDB can't magically move capacity from an idle partition to a hot one in real time (it can over time, we'll get to that). The 1,000 WCU limit is a physical constraint of the partition itself.

This is called a hot partition, and it's the most common scaling bottleneck in DynamoDB. The table-level capacity you see in the console is essentially a fiction, what matters is the distribution of your workload across partition keys.

Let me give you a concrete example. Imagine you're building a real-time analytics system that tracks page views. You decide to use the PageID as your partition key. Sounds reasonable, right? Most pages get modest traffic. But then one article goes viral on social media, and that single PageID is now receiving 5,000 writes per second. The partition holding that key can only handle 1,000 WPS, the other 4,000 writes get throttled, even though your table has 10,000 WCUs provisioned and the vast majority of that capacity is unused. The table isn't the bottleneck, the partition is.

To make this worse, there's no CloudWatch metric that directly shows you partition-level consumption (AWS added Contributor Insights to help with this, but it's not real-time and requires explicit enablement). You have to infer hot partitions from patterns in your throttling errors and your application's access patterns.

The partition distribution also explains why certain write patterns create more problems than others. If you're loading data where the partition keys are sequential or heavily time-based, like using auto-incrementing order IDs or timestamps as partition keys, you create what's called a "rolling hot partition." All your writes concentrate on the newest partition key values, hammering a single partition continuously while the others sit idle. DynamoDB can't split the load because there's only one active write boundary.

Global Secondary Index Propagation

Here's a related complexity that may catch you off guard: Global Secondary Index (GSI) throttling propagates to the base table. When you write an item, DynamoDB must update the base table and every GSI that includes that item. The total WCU cost is the sum of the base table write plus all GSI updates (each GSI write is calculated based on the projected attribute size, rounded up to 1 KB).

If a GSI is under-provisioned and can't handle the write volume, it throttles. When a GSI throttles, it blocks the base table write even if the base table has plenty of unused capacity. This is why you might see throttling on a table that appears to have headroom: you'd need to look at the GSI metrics. Tip: Always monitor ConsumedWriteCapacityUnits for every GSI, not just the base table.

DynamoDB Adaptive Capacity

DynamoDB has an internal mechanism called Adaptive Capacity that tries to handle uneven access patterns automatically. When Adaptive Capacity detects that a partition is receiving disproportionate traffic, it kicks off a process called Split-for-Heat.

DynamoDB continuously monitors partition usage. When it identifies a partition that's consistently hitting or exceeding its throughput limits, it automatically splits that partition into two new partitions. The items are redistributed based on their sort key (SK) values. This split effectively doubles the available write capacity for that segment of data: two partitions means 2,000 WCUs instead of 1,000.

DynamoDB also handles Split-for-Size. When a partition hits the 10 GB hard limit for storage, it's automatically split regardless of throughput concerns. Between these two mechanisms, DynamoDB can theoretically scale to handle any workload without manual intervention.

The problem is that this takes several minutes to complete.

In case you forgot, CloudWatch metrics are aggregated at one-minute intervals. Auto-scaling (if you're using provisioned mode) requires two consecutive minutes of exceeding the target utilization threshold before it even triggers a scaling event. Then the actual UpdateTable operation takes several minutes to complete as DynamoDB provisions additional capacity and potentially splits partitions. During all of this, your application is throttling.

Burst capacity provides some buffer here. DynamoDB banks unused throughput for up to 300 seconds. For every second your table doesn't use its full provisioned capacity, that unused capacity gets saved in a credits bucket. If you provision 1,000 WCUs but only use 500, you're banking 500 WCUs per second. When a sudden spike hits, DynamoDB can pull from this banked capacity to handle traffic that temporarily exceeds your provisioned limit. This is why you might see brief periods where consumption exceeds provisioned capacity without throttling: you're spending your burst credits. But burst capacity is designed to handle short spikes, and it runs out quickly under sustained load.

For a sustained traffic spike (a product launch, a viral moment, a coordinated attack) this lag time translates directly into dropped requests, failed transactions, and angry customers. Adaptive Capacity will eventually save you, but "eventually" can mean 5-10 minutes of degraded performance (and please tell me you're doing load shedding, otherwise this can turn into a cascading failure).

There are also failure modes where Adaptive Capacity can't help at all. If you're using a Local Secondary Index (LSI), DynamoDB can't split the partition because LSIs enforce a 10 GB item collection limit per partition key. The problem here is that Local Secondary Indexes enforce a 10 GB limit per partition key value: all items sharing a PK, plus their LSI projections, must fit within 10 GB. When this limit is approached, DynamoDB can't split the partition since that would break the LSI's contiguous storage requirement. In my experience it's very rare that you'd hit this limit, but it's important to know it's there.

Design Keys That Don't Create Heat

Before we talk about advanced patterns like write sharding, let's cover designing partition keys that naturally avoid hot spots. The fundamental principle is high cardinality with wide dispersion.

Your partition key should have many possible unique values (high cardinality), and your application's access pattern should distribute writes roughly evenly across those values (wide dispersion). If both conditions are met, DynamoDB can spread your workload across many partitions, and no single partition becomes a bottleneck.

Here are the anti-patterns that create hot partitions:

• Sequential or auto-incrementing IDs: Using ORDER#1, ORDER#2, ORDER#3 as partition keys means all new writes target the highest number, creating a rolling hot spot.

• Timestamp-based partition keys: TIMESTAMP#2025-09-29T15:30:00 suffers the same problem: writes cluster around the current timestamp.

• Low-cardinality attributes: Using Region (when you only have 3 regions) as a partition key concentrates writes on whichever region is most active.

Good partition key choices include:

• UUIDs or randomly generated IDs: USER#a7f3c891-4d2e-4b29-8f34-d9e21ba7c04f provides excellent distribution. Every user gets a unique partition key with no clustering.

• Hashed values: If you must use sequential IDs, hash them first. ORDER#{md5(order_id)} spreads sequential orders across the hash space.

• Composite keys with random components: For time-series data, append a random shard suffix: SENSOR#{sensor_id}#{random_int} (we'll explore this pattern in depth shortly).

Keep in mind that even UUIDs can create temporary hot spots if your write pattern is heavily time-based. UUIDv1 embeds timestamp information, which can cluster writes. UUIDv4 (purely random) is better for high-write scenarios.

Important: Don't use auto-incrementing integers as your partition key for high-volume writes. The newest integer value will always be the hot spot.

Here's what good partition key design looks like in code:

import uuid
import hashlib
import random

# Bad: Sequential, creates hot partition at boundary
def create_order_bad(order_counter):
    pk = f"ORDER#{order_counter}"
    return pk

# Good: Random UUID provides wide dispersion  
def create_order_good():
    pk = f"ORDER#{uuid.uuid4()}"
    return pk

# Better for time-series: Add shard suffix
def create_sensor_reading(sensor_id):
    shard = random.randint(0, 9)  # 10 shards
    pk = f"SENSOR#{sensor_id}#{shard}"
    return pk

# Good: Hash sequential IDs to spread them
def create_order_hashed(order_id):
    hash_val = hashlib.md5(str(order_id).encode()).hexdigest()[:8]
    pk = f"ORDER#{hash_val}"
    return pk

High-cardinality partition keys prevent hot spots for most workloads. But what if your use case inherently concentrates writes? What if you're tracking global counters, processing a viral post's engagement metrics, or handling a flash sale where thousands of customers are buying the same product SKU?

That's when partition key design alone isn't enough, and we need to throw some “sharding magic” into the mix.

Write Sharding in DynamoDB

If you know high-volume traffic is coming, you can't rely on Adaptive Capacity's multi-minute lag time. Ideally your keys naturally spread writes, but if they don't, you need a way to manually control how write operations are spread. That technique is called write sharding.

The idea is that instead of using a single partition key for a high-contention entity, you create N synthetic partition keys by appending a shard identifier. When writing, you randomly select one of the N shards. This spreads writes across N logical partition keys, which DynamoDB can distribute across physical partitions. It does not guarantee one physical partition or 1,000 WCUs of capacity per shard.

The tradeoff is that reads become more complex, since to retrieve the complete data for that entity you now have to query all N shards in parallel (scatter) and then aggregate the results on the client side (gather). You're gaining write throughput, but at the cost of read complexity.

Implementing Write Sharding

import random
import boto3

dynamodb = boto3.resource('dynamodb')
table = dynamodb.Table('Analytics')

SHARD_COUNT = 10  # Creates 10 logical shards; physical partition placement and capacity are not guaranteed

def increment_page_views(page_id: str, count: int = 1):
    """
    Increment page view counter using write sharding.
    Randomly selects one of N shards to distribute write load.
    """
    # Randomly select a shard (0 to SHARD_COUNT - 1)
    shard_id = random.randint(0, SHARD_COUNT - 1)
    
    # Create sharded partition key
    pk = f"PAGE#{page_id}#SHARD#{shard_id:02d}"
    
    # Atomic increment using UpdateExpression
    table.update_item(
        Key={'PK': pk, 'SK': 'VIEWS'},
        UpdateExpression='ADD view_count :inc',
        ExpressionAttributeValues={':inc': count}
    )

With this design, each write hits a random shard amongst the 10 shards that we have. If you're processing 5,000 writes per second and they're evenly distributed across 10 shards, each shard receives roughly 500 writes per second, which is hopefully manageable with the 1,000 WCUs that each partition has (remember each write consumes 1 WCU per 1 KB of data written, rounded up). If it can't, you can either aim for more shards, or rely on burst capacity (temporary use of unused table throughput, subject to partition limits) and Adaptive Capacity (using some capacity from other partitions of the same table) while you wait for Split-for-Heat (DynamoDB splitting the shard into multiple physical partitions behind the scenes) to catch up.

Scatter-Gather Query

Reading the total requires querying all shards:

import concurrent.futures
from typing import Dict

def get_page_view_total(page_id: str) -> int:
    """
    Retrieve total page views across all shards.
    Uses parallel queries (scatter) then sums results (gather).
    """
    
    def query_single_shard(shard_id: int) -> int:
        """Query one shard and return its count"""
        pk = f"PAGE#{page_id}#SHARD#{shard_id:02d}"
        
        try:
            response = table.get_item(
                Key={'PK': pk, 'SK': 'VIEWS'}
            )
            return response.get('Item', {}).get('view_count', 0)
        except Exception as e:
            # Log error but don't fail entire operation
            print(f"Error querying shard {shard_id}: {e}")
            return 0
    
    # Scatter: Execute parallel queries to all shards
    with concurrent.futures.ThreadPoolExecutor(max_workers=SHARD_COUNT) as executor:
        futures = [executor.submit(query_single_shard, i) 
                   for i in range(SHARD_COUNT)]
        results = [f.result() for f in concurrent.futures.as_completed(futures)]
    
    # Gather: Sum all shard counts
    total = sum(results)
    return total

The Economic Tradeoff of Sharding

Sharding also affects your DynamoDB bill, not just your sanity code complexity. Reading a sharded counter requires N queries instead of 1, so if you have 10 shards and each query reads 0.1 KB, you're consuming 10 RCUs instead of 1 RCU per read (assuming strongly consistent reads) even though the total data read is well below the 4 KB limit for a single RCU.

If your counter is read 1,000 times per second, sharding increases your read capacity consumption from 500 RCUs/sec to 5,000 RCUs/sec. At $0.00013 per RCU-hour in provisioned mode, that's roughly $285/month versus $28/month in read costs alone. Ouch 💸.

A counter isn't the best example, since in that case you usually pre-compute it (i.e. keep the counter as a separate entity, and update it with strong consistency every time you write new data). But overall it serves to show how much sharding can cost you. Still, it's usually better than getting throttled!

Choosing the Right Shard Count

For most use cases, I suggest you start with N=10 shards. As you get traffic, monitor your CloudWatch metrics (specifically WriteThrottleEvents for write throttling and ConsumedWriteCapacityUnits per shard if you've instrumented that). If you're still seeing throttles, increase N. If your read performance is suffering and throttling is rare, you might be over-sharded and can reduce N. The caveat here is that increasing N is pretty simple: You just change the value in the code and start writing to new shards 11, 12, etc. To decrease N you'll have to rewrite every single entity, since you'll need to replace the value of {shard} in the PK ENTITY#{shard}. And you can't just overwrite that field, PKs are read-only, so you need to atomically create a new object and delete the old one. The script is pretty easy, but it's 1 read, 1 write and 1 delete in a transaction per entity in your table. Ouch again 💸💸.

One more consideration: you don't have to shard your entire table. Shard only the hot entities. If you have a leaderboard table tracking scores for millions of users, but only the top 100 users receive heavy write traffic, shard those 100 partition keys. The rest can use normal, single-partition-key design.

Conclusion

Your table capacity doesn't matter if one partition key gets all the traffic. Physical partitions max out at 1,000 WCUs, regardless of table capacity. Adaptive capacity can rebalance throughput within table and partition limits; sudden spikes still require workload-aware planning.

The first step is to design partitions well. This is the most critical thing in DynamoDB, actually, so I suggest you read on DynamoDB Design and DynamoDB Transactions. Understanding How DynamoDB Scales is also important. Then, if you need to prepare for traffic spikes, shard your entities.

DynamoDB scales incredibly well, but only if you design for how it actually works, not how you wish it worked (or worse, for how relational databases work). Which is why I somehow end up writing so much about DynamoDB.

Aurora DSQL is the subject of this June 2025 launch-era review. Historical note (September 2026): Aurora DSQL now supports foreign keys, sequences and identity columns. The compatibility and migration discussion below reflects the original release, and the code examples still need a coordinated refresh. Marketed as the “Fastest serverless distributed SQL database for always available applications“, it promises PostgreSQL compatibility with virtually unlimited scale. But here's what AWS doesn't tell you upfront: it's going to break your existing PostgreSQL application. Not maybe. Not sometimes. It will break it (and that's actually by design).

I've been really excited about this since I heard about it at re:Invent 2024, but my reservations started when I talked to the team behind it and they told me there are no Foreign Keys. So now that it's on General Availability, I decided to spend a bit of time diving deep into Aurora DSQL and discovering all the ways this "compatible" database service differs from what you'd expect. The good news? Once you understand what you're dealing with, DSQL becomes incredibly powerful. The bad news? You're going to need to rethink how you build database applications. It's true, there are no Foreign Keys.

What Aurora DSQL Actually Is

Aurora DSQL isn't just Aurora with auto-scaling bolted on (that would be too easy). It's a fundamentally different beast that happens to speak PostgreSQL's language. Think of it as a distributed, serverless SQL database that borrowed PostgreSQL's syntax but threw out most of the operational model you're used to.

The architecture consists of four main layers that scale independently:

• A relay and connectivity layer handling client connections.

• Compute and database engines processing your queries.

• A distributed transaction log managing concurrency and isolation.

• A distributed storage layer.

A control plane orchestrates everything across three Availability Zones, automatically scaling each component based on demand (which sounds great until you see the DPU bills).

Here's where it gets interesting: DSQL uses Optimistic Concurrency Control instead of traditional locking. No more deadlocks, but also no more assuming your transaction will just work. Every write operation can fail at commit time if another transaction touched the same data, and your application needs to handle that gracefully. This isn't a edge case, it's normal operation.

The service comes in two flavors: single-Region clusters that operate active-active across three AZs within one AWS Region (meaning it's highly available out of the box), and multi-Region clusters that synchronously replicate across multiple regions with a "witness" region handling log consensus (much better than Aurora Global Databases). The single-Region setup targets 99.99% availability, while multi-Region promises 99.999% (those extra nines cost extra, obviously).

What makes this serverless is that you don't provision instances or manage scaling policies. DSQL automatically partitions your data based on primary key ranges and access patterns, spins up compute resources as needed, and bills you in Distributed Processing Units (DPUs) for actual usage. It's infrastructure abstraction taken to its logical extreme (for better and worse). At least it's truly truly serverless, not like Aurora Serverless.

The Three Things That Will Surprise You

1. Optimistic Concurrency Control Will Change How You Code

Forget everything you know about database transactions. DSQL doesn't use locks during transaction execution, it validates conflicts only at commit time. This means any write transaction can fail with a serialization error, even if it ran perfectly during execution.

When conflicts occur, you'll see PostgreSQL serialization errors with SQLSTATE '40001' or DSQL-specific 'OC000' and 'OC001' error codes. Your application must catch these and retry the entire transaction. Not some transactions. All transactions. Here's what robust retry logic looks like:

public void executeWithRetry(Runnable transactionLogic) {
    int maxRetries = 3;
    int attempt = 0;
    
    while (attempt < maxRetries) {
        try {
            transactionLogic.run();
            return; // Success
        } catch (SQLException e) {
            if (e.getSQLState().startsWith("OC") || 
                e.getSQLState().equals("40001")) {
                attempt++;
                if (attempt >= maxRetries) throw e;
                
                // Exponential backoff with jitter
                Thread.sleep((long) (Math.pow(2, attempt) * 100 + Math.random() * 100));
            } else {
                throw e; // Non-retryable error
            }
        }
    }
}

The key insight: every operation must be idempotent. You can't just increment a counter and retry, you need to structure operations as "set counter to X if current value is Y" patterns. This fundamentally changes how you design business logic (trust me, it's a bigger shift than it sounds).

2. PostgreSQL "Compatibility" Has Some Creative Interpretations

DSQL speaks PostgreSQL's wire protocol and supports a substantial subset of SQL, but "compatible" doesn't mean "identical". The list of unsupported features reads like a PostgreSQL greatest hits album: no foreign keys, no triggers, no sequences, no PL/pgSQL, no user-defined functions except pure SQL ones. To be honest I can personally do without all of those, except foreign keys, that's the one thing I miss.

The isolation level is fixed at Repeatable Read (you can't change it), transactions have size limits of 10 MiB and 3,000 row modifications, and you can't mix DDL and DML in the same transaction. Each transaction can contain exactly one DDL statement, period.

Here's where it gets tricky for migrations. Take this seemingly innocent PostgreSQL table:

-- This works in PostgreSQL
CREATE TABLE orders (
    id SERIAL PRIMARY KEY,
    customer_id INTEGER REFERENCES customers(id),
    total NUMERIC(10,2),
    created_at TIMESTAMP DEFAULT NOW()
);

CREATE TRIGGER update_customer_stats 
    AFTER INSERT ON orders 
    FOR EACH ROW 
    EXECUTE FUNCTION update_customer_totals();

In DSQL, you'd need to rewrite this as:

-- DSQL version requires major changes
CREATE TABLE orders (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    customer_id UUID,  -- No foreign key constraint
    total FLOAT,       -- NUMERIC not supported in keys
    created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW()
);

-- Trigger logic moves to application layer (as it should!!)
-- Foreign key validation becomes application responsibility (this is annoying, but it's manageable)

The absence of foreign keys means referential integrity becomes your application's job. No more relying on the database to prevent orphaned records (we've all been there, and it's not fun to debug in production).

The trick here is that foreign keys do exist in your data model, they're just no longer enforced by the database engine. You can still write relationships in your code, but if you want them enforced and want to catch the related bugs, you must do that at the application layer. Give it a long, hard thought, and you'll see that it's not the end of the world, just a quality of life tradeoff. But seriously, give it some thought. It took me 20 minutes of conversation to come to this conclusion.

3. DDL Operations Are Distributed Transactions (With Consequences)

Schema changes in DSQL aren't simple metadata updates, they're strongly consistent distributed transactions across all cluster nodes. This has profound implications for how you manage schema migrations.

Every DDL operation runs as a distributed transaction, which means it can fail due to concurrency conflicts just like regular transactions. You might see errors like "ERROR: schema has been updated by another transaction, retry as needed (OC001)" during migrations.

The restrictions are strict: one DDL statement per transaction, no mixing DDL with DML, and schema changes can take time to propagate. But here's the superpower: CREATE INDEX ASYNC runs in the background without locking your table.

-- This runs immediately and returns a job_id
CREATE INDEX ASYNC idx_orders_customer 
ON orders(customer_id);

-- Monitor progress
SELECT * FROM sys.jobs WHERE job_id = 'your_job_id';

-- Wait for completion in scripts
SELECT sys.wait_for_job('your_job_id');

Unlike PostgreSQL where CREATE INDEX can lock your table for hours on large datasets, DSQL's async approach lets you add indexes to active tables without downtime. This is genuinely game-changing for large-scale applications (just don't forget to check the job status).

Getting Aurora DSQL Actually Working

Let's get our hands dirty with real setup, because since I spent time researching this I've decided you need to spend time on this as well. Also, it's important. DSQL requires IAM authentication (no traditional passwords), SSL connections (non-negotiable), and specific client configurations that differ from standard PostgreSQL.

Prerequisites and IAM Setup

First, you need IAM permissions for both cluster management and database connections. Here's the minimum policy for getting started:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "dsql:CreateCluster",
                "dsql:GetCluster",
                "dsql:ListClusters",
                "dsql:DbConnectAdmin"
            ],
            "Resource": "*"
        }
    ]
}

For production environments, you'll also need customer-managed KMS keys and VPC endpoint permissions, but this gets you connected (security hardening comes later).

Creating Your First Cluster

Single-region cluster creation is straightforward via the AWS CLI:

# Create a basic single-region cluster
aws dsql create-cluster \
    --region us-east-1 \
    --deletion-protection-enabled \
    --tags Key=Environment,Value=development

# For multi-region (requires coordination)
aws dsql create-cluster \
    --region us-east-1 \
    --multi-region-properties '{
        "witnessRegion": "us-west-2",
        "clusters": ["arn:aws:dsql:us-east-2:account:cluster/peer-cluster"]
    }'

The cluster creation returns an ARN and endpoint URL. Save that endpoint, you'll need it for every connection (and it's not changing like RDS endpoints do during failovers).

Connecting to DSQL Using IAM Authentication

DSQL connections require IAM tokens instead of passwords, and these tokens have dual expiry mechanics that will trip you up if you haven't used them before. The token expires (default 15 minutes, maximum 1 week), but the database connection also expires after one hour regardless of token validity.

Generate tokens using the AWS CLI:

# Generate admin token valid for 1 hour
aws dsql generate-db-connect-admin-auth-token \
    --hostname your-cluster-endpoint.dsql.amazonaws.com \
    --region us-east-1 \
    --expires-in 3600

# Generate token for specific database role
aws dsql generate-db-connect-auth-token \
    --hostname your-cluster-endpoint.dsql.amazonaws.com \
    --region us-east-1 \
    --username myuser \
    --expires-in 3600

Now connect with psql (note the required SSL mode):

PGSSLMODE=require psql \
    --dbname postgres \
    --username admin \
    --host your-cluster-endpoint.dsql.amazonaws.com \
    --password  # Paste the IAM token when prompted

For application connections, here's a Python example using psycopg:

import boto3
import psycopg
from urllib.parse import quote_plus

def get_dsql_connection():
    client = boto3.client('dsql', region_name='us-east-1')
    
    token = client.generate_db_connect_admin_auth_token(
        hostname='your-cluster-endpoint.dsql.amazonaws.com',
        expiresIn=3600
    )
    
    conn_string = (
        f"postgresql://admin:{quote_plus(token)}@"
        f"your-cluster-endpoint.dsql.amazonaws.com:5432/postgres"
        f"?sslmode=require"
    )
    
    return psycopg.connect(conn_string)

Still with me? The connection complexity is real, but it's the price of IAM integration (and honestly it's more secure than password-based auth once you get the patterns down).

Schema Design Patterns

Once connected, your first decision is primary key strategy. For write-heavy tables, avoid sequential integers like the plague, they create hotspots in DSQL's distributed storage:

-- Good: Random distribution for high-write tables
CREATE TABLE events (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    user_id UUID,
    event_type VARCHAR(50),
    data JSONB,
    created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW()
);

-- Bad: Sequential keys create write hotspots
CREATE TABLE events_bad (
    id SERIAL PRIMARY KEY,  -- This will bottleneck
    user_id INTEGER,
    event_type VARCHAR(50),
    data JSONB,
    created_at TIMESTAMP
);

Create indexes asynchronously to avoid blocking operations:

-- Start async index creation
CREATE INDEX ASYNC idx_events_user_type 
ON events(user_id, event_type);

-- Check progress
SELECT job_id, status, progress_percentage 
FROM sys.jobs 
WHERE operation_type = 'CREATE_INDEX';

Key sizing limits apply: primary keys can't exceed 1 KiB total size, individual columns in keys max out at 255 bytes, and primary keys can include up to 8 columns. Plan accordingly (VARCHAR(320) email fields won't work in composite keys).

Where Aurora DSQL Gets Expensive (And How to Avoid It)

DSQL's pricing model using Distributed Processing Units (DPUs) can surprise you. Unlike provisioned databases where you pay for capacity whether you use it or not, DSQL bills for actual computation, I/O, and storage operations. This sounds great until you realize inefficient queries now directly impact your AWS bill instead of just wasting already wasted capacity.

Understanding DPU Consumption Patterns

DPUs measure normalized compute units across different operations. Here's what drives costs:

• WriteDPU: Write operations including inserts, updates, deletes

• ReadDPU: Query processing for SELECT operations

• ComputeDPU: CPU-intensive operations like complex joins and aggregations

• MultiRegionWriteDPU: Additional overhead for multi-region write replication

Monitor these with CloudWatch:

# Set up CloudWatch alarms for DPU spikes
aws cloudwatch put-metric-alarm \
    --alarm-name "DSQL-HighDPU" \
    --alarm-description "Aurora DSQL high DPU usage" \
    --metric-name TotalDPU \
    --namespace AWS/AuroraDSQL \
    --statistic Average \
    --period 300 \
    --threshold 1000 \
    --comparison-operator GreaterThanThreshold \
    --dimensions Name=ResourceId,Value=your-cluster-id

A single inefficient query scanning millions of rows can generate hundreds of DPUs in minutes. Query optimization isn't just about performance anymore, it's about keeping your AWS bill reasonable (ask me how I learned this).

Primary Key Design Impact on Performance and Cost

Remember how I mentioned UUIDs for write-heavy tables? Here's why it matters for your wallet. Sequential primary keys concentrate writes on single partitions (remember that the D in DSQL stands for Distributed), forcing DSQL to scale those specific partitions while others sit idle. This inefficient resource utilization shows up in your DPU costs.

-- This table design will cost you more
CREATE TABLE audit_log (
    id BIGINT PRIMARY KEY,  -- Sequential writes = hotspot
    action VARCHAR(100),
    timestamp TIMESTAMP,
    user_id UUID
);

-- This distributes load and costs
CREATE TABLE audit_log_optimized (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),  -- Distributes writes
    action VARCHAR(100),
    timestamp TIMESTAMP,
    user_id UUID
);

-- Create covering index for common queries
CREATE INDEX ASYNC idx_audit_user_time 
ON audit_log_optimized(user_id, timestamp);

The covering index is an additional trick that prevents expensive primary key lookups for common query patterns, reducing ReadDPUs significantly.

Connection Pools in Aurora DSQL

Traditional connection pools assume connections stay valid until explicitly closed or the database terminates them. DSQL breaks this assumption with dual expiry: IAM tokens expire, and connections automatically terminate after one hour.

Your connection pool needs to handle both scenarios:

import time
from dataclasses import dataclass
from typing import Optional

@dataclass
class DSQLConnection:
    connection: psycopg.Connection
    created_at: float
    token_expires_at: float
    
    def is_valid(self) -> bool:
        now = time.time()
        # Check both token expiry and DSQL's 1-hour limit
        return (now < self.token_expires_at and 
                now < self.created_at + 3540)  # 59 minutes

class DSQLConnectionPool:
    def __init__(self, min_connections=2, max_connections=10):
        self.min_connections = min_connections
        self.max_connections = max_connections
        self.connections = []
        self.dsql_client = boto3.client('dsql')
    
    def get_connection(self) -> psycopg.Connection:
        # Remove expired connections
        self.connections = [c for c in self.connections if c.is_valid()]
        
        if not self.connections:
            return self._create_new_connection()
        
        return self.connections.pop().connection
    
    def _create_new_connection(self) -> psycopg.Connection:
        token = self.dsql_client.generate_db_connect_admin_auth_token(
            hostname='your-cluster-endpoint.dsql.amazonaws.com',
            expiresIn=3600  # 1 hour
        )
        
        conn = psycopg.connect(
            f"postgresql://admin:{quote_plus(token)}@"
            f"your-cluster-endpoint.dsql.amazonaws.com:5432/postgres"
            f"?sslmode=require"
        )
        
        return conn

This complexity is manageable with the right abstractions, but it's definitely not plug-and-play with existing connection pooling libraries (most will need configuration adjustments at minimum).

Query Optimization for OCC and Transaction Limits

DSQL's transaction limits (10 MiB data modified, 3,000 rows affected) require batching strategies for large operations:

def batch_insert_with_retry(records, batch_size=1000):
    for i in range(0, len(records), batch_size):
        batch = records[i:i+batch_size]
        
        execute_with_retry(lambda: insert_batch(batch))

def insert_batch(records):
    with get_connection() as conn:
        with conn.cursor() as cur:
            # Use COPY for efficient bulk inserts
            with cur.copy("COPY events (user_id, event_type, data) FROM STDIN") as copy:
                for record in records:
                    copy.write_row([record.user_id, record.event_type, record.data])

The COPY command is significantly more DPU-efficient than individual INSERT statements for bulk operations, but you still need to respect the transaction size limits.

The Missing Pieces You Need to Know

Let's talk about the operational aspects AWS doesn't emphasize in the marketing materials. DSQL is powerful, but it requires new operational patterns and monitoring approaches.

Monitoring and Alerting That Actually Matters

Standard PostgreSQL monitoring tools won't help you with DSQL 🫠. You need CloudWatch dashboards focused on DSQL-specific metrics:

# Create custom CloudWatch dashboard
aws cloudwatch put-dashboard \
    --dashboard-name "Aurora-DSQL-Operations" \
    --dashboard-body '{
        "widgets": [
            {
                "type": "metric",
                "properties": {
                    "metrics": [
                        ["AWS/AuroraDSQL", "TotalDPU", "ResourceId", "your-cluster-id"],
                        ["AWS/AuroraDSQL", "OccConflicts", "ResourceId", "your-cluster-id"],
                        ["AWS/AuroraDSQL", "QueryTimeouts", "ResourceId", "your-cluster-id"]
                    ],
                    "period": 300,
                    "stat": "Average",
                    "region": "us-east-1",
                    "title": "DSQL Performance Metrics"
                }
            }
        ]
    }'

Critical metrics to monitor:

• OccConflicts: High values indicate data contention requiring application logic review

• QueryTimeouts: Suggests queries hitting DSQL's transaction time limits

• TotalDPU: Your primary cost indicator and performance baseline

• CommitLatency: Increased latency can indicate replication or storage issues

Set aggressive alarms on these metrics. OCC conflicts above 5% of total transactions usually indicate design problems that will only get worse under load.

Backup and Recovery Strategy (It's Different)

DSQL integrates with AWS Backup, not native database snapshots. This means backup policies, retention, and restoration work through the AWS Backup service:

# Create backup plan for DSQL
aws backup create-backup-plan \
    --backup-plan '{
        "BackupPlanName": "dsql-daily-backups",
        "Rules": [{
            "RuleName": "daily-backup",
            "TargetBackupVault": "default",
            "ScheduleExpression": "cron(0 2 * * ? *)",
            "Lifecycle": {
                "DeleteAfterDays": 30
            }
        }]
    }'

Multi-region cluster restoration requires coordination between AWS Backup and DSQL APIs, and it's currently only supported via CLI (not the console). Plan your disaster recovery procedures accordingly, and test them. The restore process creates new clusters rather than in-place restoration, so test a lot.

Security Configuration for Production

VPC endpoints are essential for production deployments of anything that doesn't live in your VPCs, and Aurora DSQL (unlike regular Aurora) doesn't go into your VPC. DSQL requires separate endpoints for management operations and database connections:

# Create VPC endpoint for database connections
aws ec2 create-vpc-endpoint \
    --vpc-id vpc-12345678 \
    --service-name $(aws dsql get-vpc-endpoint-service-name \
                    --cluster-identifier your-cluster-arn \
                    --query ServiceName --output text) \
    --vpc-endpoint-type Interface \
    --subnet-ids subnet-12345678 subnet-87654321 \
    --security-group-ids sg-12345678 \
    --private-dns-enabled

Migration Planning Reality Check

Here's the hard truth: DSQL isn't a drop-in replacement for PostgreSQL, despite the compatibility marketing. Successful migrations require systematic assessment and application refactoring.

Assessment framework:

1. Dependency Analysis: Catalog usage of unsupported features (foreign keys, triggers, sequences, PL/pgSQL)

2. Transaction Pattern Review: Identify long-running transactions, mixed DDL/DML operations, large batch operations

3. Query Complexity Evaluation: Find queries that might hit transaction time limits or benefit from different indexing strategies

4. Connection Management Audit: Assess current pooling strategies and connection lifecycle management

For applications heavily dependent on PostgreSQL-specific features, consider DSQL for new microservices rather than migrating existing monoliths. The serverless scaling and operational simplicity make it excellent for greenfield projects designed with its constraints in mind.

The learning curve is real, but once you adjust to the OCC model and embrace the serverless operational paradigm, DSQL delivers on its scalability promises. Just don't expect it to be PostgreSQL with better marketing (because it's not, and that's actually the point).

This is part 4 of this Microservices vs Agentic AI series. In the first two parts of this series we laid out the core distinctions between Microservice and Agentic AI architectures. Part 1 traced their separate reasons for being: Microservices emerged to solve the challenges of monolithic scale by decomposing applications along business domain lines. Agentic AI, on the other hand, leverages the recent power of Large Language Models (LLMs) to tackle complex tasks through autonomous reasoning and functional decomposition. This led us to see how their fundamental approaches to autonomy and specialization naturally diverge.

Part 2 then took us into the thick of their runtime behaviors. We contrasted the predictable, deterministic logic common in Microservice communication and state management (using Sagas for eventual consistency) with the intelligent, context-rich flows, operational memory, and inherent non-determinism that characterize Agentic AI systems and their unique pattern needs (and the unique challenges they bring).

Following that, Part 3 led us deep into the operational realities. We considered the demands and challenges of operating agentic systems, compared to the more understood load-based, horizontal scaling of microservices. We discussed the distinct resilience challenges posed by infrastructure failures versus cognitive ones, the critical differences in observability needs (seeing the "why" of AI decisions), and the current state of tooling and the MLOps/LLMOps frontier. We also briefly touched upon the cost structures of both architectures, with a focus on understanding LLM token consumption.

Now that we've established this comprehensive picture of their individual characteristics, from foundational philosophy through runtime execution to operational demands, I'd like to propose something different for this fourth and final article on this series. We're going to explore how these two architectural paradigms can be combined, and what that means for system design. Beyond the technical integration, we'll examine the mindset shifts required from the teams involved. And finally, we'll look towards future considerations, especially the continued operational importance of security and ethics. The goal is to put everything together and offer a complete perspective. Let's hope I can pull that off.

Hybrid Architecture Patterns in Practice

Let me kick this off by saying I am not recommending you build a hybrid system with both microservices and AI agents just so you can “get the benefits of both patterns” or something like that. That's the completely wrong approach to architecture. I'm not saying either that you should never combine them. The purpose of this section is to highlight the areas to pay special attention to if you ever find the need to add agentic capabilities to some parts of a microservices architecture, or vice versa. Basically, to warn you of where things will likely get extra difficult, the points where the complexities of each pattern overlap.

Common Integration Patterns

These are the most obvious ways in which these two patterns would combine:

• Agent as Intelligent Facade: The agent provides a natural language (or other intuitive) interface, understanding user intent and translating it into calls to existing backend microservice APIs. Example: A user asks the OmniMart chatbot, "Can I get free shipping on order 12345 to my alternate address?". The agent parses this, calls the Order Service API for order details, the Customer Profile Service API for addresses, and potentially a Shipping Policy Service API (or consults a KB), synthesizes the answer, and responds naturally. Basically the agent is the user interface, and the microservices are called as tools by that agent instead of (or in addition to) as a backend by a typical frontend.

• Microservice as Reliable Tool: Agents offload complex, sensitive, or mission-critical business logic execution to dedicated microservices, treating them as reliable tools. Example: The OmniMart financial planning agent uses a validated PortfolioRiskCalculationService microservice instead of trying to implement complex financial math via prompts. The agent focuses on goal setting and strategy, the microservice provides reliable calculation. This looks very similar to the above, but the core distinction is that in Agent as Intelligent Facade you start from the actions (microservices) and then expose them via an agent. In Microservice as Reliable Tool the core of the system is the agent, and microservices just solve things that LLMs are bad at.

• Event-Driven Trigger: Microservices emit business events (e.g., using Amazon EventBridge) that trigger agentic processes for deeper analysis or complex follow-up actions. Example: OmniMart Fraud Detection Service (a microservice using programmatic rules) detects a potentially suspicious pattern and emits a MediumRiskTransactionEvent. An Investigation Agent subscribed to the topic receives the event, then uses tools to pull data from multiple other services (user history, location data, etc.), performs more complex reasoning, and decides whether to escalate to a human analyst or automatically block the transaction via another tool call. This way we'd be using agents and microservices interchangeably, where components of an Event-Driven Architecture emit events and those are handled by other components, and any component can be a microservice or an agent. One thing to keep in mind is that agents are non-deterministic, and that will complicate distributed transactions.

• Agent Orchestrating Microservices: An agent's plan involves coordinating a sequence of actions across multiple microservices. Example: The OmniMart returns agent, after interacting with the customer and approving an exception based on complex logic, calls the Order Service API (tool) to mark the item for return and the Logistics Service API (tool) to schedule a pickup. This is pretty similar to Microservice as Reliable Tool, but the microservices aren't just potentially useful tools for the agent, the tools are the end goal and the purpose of the agent is to orchestrate them. It would be like a very smart orchestrator pattern.

Challenges

While all of the above sounds like a great idea to implement, it comes with a few operational challenges:

• End-to-End Observability: Tracing a logical operation becomes significantly harder when it crosses between the agent's reasoning steps (often opaque) and multiple microservice calls (potentially asynchronous). Correlating logs and metrics across these two types of components requires meticulous instrumentation, context propagation (passing trace IDs), and potentially specialized tooling beyond standard distributed tracing offered by AWS X-Ray alone. Debugging failures that span the boundaries of microservices and agents are going to be particularly challenging, even more so than failures that just span multiple microservices.

• Security Context Propagation: How is the user's identity or authorization context securely passed from the initial interaction with the agent through its subsequent calls to backend microservices? Mechanisms like secure token exchange or carefully managed IAM permissions for agent tools are needed, adding complexity. The key problem here is that, as I've said many times in this series, agentic AI is still a new thing, and we don't have the generally accepted patterns and shared best practices that we can count on for microservices.

• Deployment Dependencies & Contracts: Changes to a microservice API used as a tool directly impact the agent. This requires careful versioning, communication between teams, and potentially updating the agent's tool description, prompts, or even retraining its ability to use the tool correctly. The annoying part here is that changes that don't add capabilities, e.g. a reduction in latency, would typically not affect the caller of a microservice, but if the caller is an AI agent then these aspects matter in tool selection, and need to be reflected in the tool's description. We've inadvertently introduced more coupling than what we're used to when dealing with APIs, and broken a fundamental tenet of microservices: callers only depend on the exposed interfaces and contracts, not on implementation details.

• Consistency Management: If an agent orchestrates actions that modify state across multiple microservices, ensuring overall transactional integrity becomes very challenging. Implementing a Saga pattern driven by the agent's logic is complex and potentially fragile compared to service-level Sagas (remember that LLMs are non-deterministic!). Careful design is needed to avoid leaving backend systems in inconsistent states if the agent's workflow fails mid-process. To be fair, a perfect distributed system would already be designed to recover from any kind of failure, so we already know how to handle this. But of course, no system is really perfect, and those imperfections might become more evident with LLM hallucinations having a much higher probability of occurrence than network failures.

People and Process: The Socio-Technical Shift

This section is intended to help those who already work with microservices and would like to add AI capabilities to your software. It's based on my experiences at Caylent building really interesting Generative AI applications, and a few discussions with some friends who tried to build GenAI teams.

Evolving Team Structures and Skillsets

Microservice development often relies on cross-functional teams blending at least backend and infrastructure, and ideally with some decent domain expertise. Building sophisticated agentic systems requires a broader skillset across the team:

• AI/ML Expertise: The obvious skills here are model understanding, evaluation and optimization. But the real AI/ML experts (not just GenAI experts) also bring a lot of relevant experience. Remember that GenAI and LLMs are just a special flavor of ML, and while the uses are different, a lot of the building blocks are similar enough. Look out especially for MLOps expertise, and knowledge of how to prepare and manage data. If you get LLMOps, you've hit the jackpot.

• Prompt Engineering: This sounds completely trivial now that we have reasoning models, and fortunately that ridiculous hype of “everyone will just be a prompt engineer” died really quickly. You don't need a prompt engineer. But you do need someone to understand how prompting works, including prompt caching, cost optimization, and other stuff related to prompts. At least one person in the team should have a good idea about this.

• Software and Cloud Engineering: Do not try to build software without software engineers, and do not deploy to the cloud without cloud engineering. I know a lot of brilliant people who I'm pretty sure could code an LLM from scratch, but who don't know how AWS Lambda works or how to achieve high availability and scalability. Generative AI applications need generative AI skills, but they also need application skills.

• Data Science/Engineering: This is a bit more situational, and will depend on the exact skillset of your AI/ML experts and how you manage data. Creating and using a Knowledge Base should be easy enough with GenAI expertise, even the very fine details like choosing the embeddings size, optimizing the vector database, or fine-tuning the embeddings model. However, if you need to do some data processing, you'll need someone with data science and/or data engineering skills. Any cloud engineer can stand up a data pipeline on AWS, and any software engineer can write the ETL code (it's just the MapReduce pattern). But without the right conceptual and domain knowledge you'll likely be processing useless data and reaching wrong conclusions.

• Domain Expertise: This is always super important. But in my experience, it's even more important for Generative AI applications. We're collectively still discovering where GenAI is most useful, and where it sounds useful on paper but isn't reliable or cheap enough yet. I work at a consulting company, so helping companies figure out the business impact of our work is second nature at this point. But I've found that while building blindly is always bad, building GenAI stuff blindly is even worse than usual. Btw I dumped a lot of thoughts on this in the 2025 Outlook on GenAI whitepaper I co-wrote with our head of AI.

• UI/UX Design: In many cases, AI is redefining the way users interact with software. In many cases we've traded screens filled with buttons for a chatbot that provides easier access to the same functionality. Sometimes that's not the best decision. On some occasions you need a combination of both, with either a dashboard on screen and a chat to ask more questions, or a chatbot that can output graphs instead of just text. Sometimes we're just making backend changes but the whole expectation shifts a bit (remember non-determinism!!). And sometimes we want to do away with screens entirely, e.g. with Speech-to-Speech models. There are no clear patterns yet, so you need someone capable of understanding and especially designing how users interact.

Smaller teams will have a hard time including all the skills I listed above, and you're going to need to make compromises. When you do, be aware of what you'll need and why, and I'd suggest don't hesitate to rely on consultants or fractional resources, either for fewer hours a day or for just a few weeks.

Organizations that need to field teams like this multiple times will likely gravitate towards centralized expertise, just like we already do with cloud or security: Cloud Center of Excellence, Security Operations Center, platform engineering with best practices. I believe that's a good investment for the long term. However, for the shorter term the practice that I've seen have the biggest impact is to create a culture of sharing knowledge. Many times an engineer with some knowledge can get you 80% of the way there, and you can bring in the experts for a few hours to help push the rest of the way. Moreover, sharing the lessons from different projects helps everyone grow. So right now I'd put my focus there: Build cross-functional teams as good as you can, and create a culture of sharing knowledge across the entire company.

Conclusion

And with that, our deep dive comparison into Microservice and Agentic AI architectures comes to an end. Across these four parts, we've journeyed from their foundational principles and distinct origins (Part 1), through their contrasting runtime dynamics concerning communication, state, and predictability (Part 2), into the practical operational realities of scale, resilience, observability, tooling, deployment, and cost (Part 3), and finally, explored the integration points and challenges in hybrid systems and what all of this means for the humans involved (Part 4, this one).

While both architectural styles leverage decomposition and operate as distributed systems, they are fundamentally different tools designed for different primary purposes. Microservices offer a mature, robust paradigm for structuring large applications around business domains, optimizing for engineering lifecycle agility, operational scalability, and reliability through well-understood patterns and a rich tooling ecosystem. Agentic AI provides a powerful, though operationally less mature, paradigm for automating complex tasks, enabling autonomous reasoning and action, and creating intelligent interactions by orchestrating LLMs, tools, and knowledge.

This series was never intended as a guide for choosing one over the other as if they were direct competitors for every problem. My initial goal was to uncover useful patterns and lessons from microservices that could be applied to Agentic AI, under the assumption that since both patterns are based on decomposition, there would be a lot of similarities (in fact, I said multiple times that agents were just LLM-based microservices). I've failed at that goal: I found a lot more differences than similarities, and few transferrable lessons.

But even if I failed at my original goal, I hope exploring both patterns, with their respective strengths, weaknesses, unique characteristics, and operational demands, still yielded some value to you. My hope is that understanding these nuances allows you to avoid misapplying principles from one domain to the other, to set realistic expectations, and, most importantly, to identify opportunities where their strengths can be combined.

As LLMs keep percolating the software world, I don't think I'm too crazy if I predict that the future of many complex systems will likely be hybrid, blending the deterministic reliability of microservices for core business logic and data management with the adaptive intelligence of agents for sophisticated interaction, automation, and reasoning. So this should all be useful as we move towards that future.

In the first two parts of this series we established a clear picture of the fundamental differences between Microservice and Agentic AI architectures. Part 1 traced their origins and motivations, revealing how Microservices evolved to solve software lifecycle and scaling challenges by decomposing applications along business domain lines, while Agentic AI leverages Large Language Model (LLM) breakthroughs to automate complex tasks through autonomous reasoning and action. Part 2 explored the consequences for runtime behavior, contrasting Microservices' reliance on defined APIs, "dumb pipes," deterministic logic, and established patterns for eventual data consistency with Agentic AI's use of intelligent flows, contextual memory, cognitive reliability patterns, and inherent non-determinism.

These foundational and runtime differences inevitably lead to distinct operational challenges and realities. Now, in this third part, we're going to dive into the operations side: How do we effectively operate, scale, and ensure the resilience of these systems in production? What are the real-world complexities surrounding observability, tooling, deployment, and lifecycle management? How do their cost structures differ, and what optimization strategies are available, particularly on AWS? Our goal remains not to prescribe one architecture over the other, but to understand the operational landscape of each, learning from their contrasts to make better-informed design and operational decisions. And to have fun while thinking about complex stuff.

Scale and Resilience Implications

Operating distributed systems at scale always involves managing scalability and resilience, but the specifics differ significantly. Let's use this as the first angle of comparison in this part.

Complexities of Scaling AI Agents

Microservice scalability requires good engineering, but it follows relatively well-understood operational patterns. The primary goal is handling request load for specific services, which involves configuring horizontal scaling mechanisms. On AWS, this means setting up Auto Scaling Groups for EC2 instances, configuring Service Auto Scaling for Amazon ECS tasks, or Kubernetes Horizontal Pod Autoscaler for EKS deployments alongside Cluster Autoscaler or Karpenter for node capacity, or relying on the inherent concurrency scaling of AWS Lambda. Monitoring focuses on metrics like CPU utilization, memory usage, request counts, or queue lengths to trigger scaling events. And yeah, this is all more or less automated, but we still need to understand the fine details.

Agentic AI scaling presents a similar picture, even if the names are different. Teams must monitor and manage capacity across multiple components. The main difference is tool maturity. Here are the main parts you need to scale:

• Foundation Model (FM) Throughput: The core reasoning engine itself can become a bottleneck. Ensuring adequate throughput and acceptable latency for LLM inference (e.g., via Amazon Bedrock or Amazon SageMaker endpoints) is critical. For example you can use Amazon Bedrock Provisioned Throughput, where you purchase dedicated inference capacity in model units, billed hourly for supported models and potentially achieve lower per-token costs at scale, but this requires accurate capacity planning and commitment. This part is very similar to scaling microservices, but the tooling isn't as mature yet: You get either self-hosted models e.g. in SageMaker (which is comparable to EC2) or a very black box service like Bedrock or external APIs (comparable to Lambda).

• Tool Scalability: Each external Tool an agent calls (often an AWS Lambda function or another API) must scale independently to handle the aggregated load from potentially many concurrent agent instances. For this part you can directly use the same techniques as with microservices (in fact, as we'll see later in this article, Tools can easily be microservices). Unless you're consuming external tools, but in that case it's the same as when a microservice uses one.

• Knowledge Base (KB) Scalability: For agents using RAG, the vector database (e.g., Amazon OpenSearch Service, Amazon Aurora with pgvector) must handle the concurrent query load generated during the retrieval step. This requires appropriate provisioning (instance sizes, shard counts for OpenSearch) or leveraging serverless options like OpenSearch Serverless or Aurora Serverless and monitoring query latency and resource utilization. You should treat this part just like scaling a regular database (after all, this is a regular database).

• Orchestration Layer Scalability: The platform or framework managing the agent's execution flow or multi-agent coordination must also scale. For managed platforms like Bedrock Agents, AWS handles this scaling automatically, though it's always important to understand the limits. For custom frameworks deployed on compute like ECS or Lambda, standard application scaling techniques apply but need to account for the specific load patterns of agent orchestration (take into special account the time waiting for inference).

Operationally, managing agentic scaling requires a more complex baseline: you can't just track overall request and database loads, at a minimum you'll need to look at FM utilization, tool performance, KB query latency, and the health of the orchestration layer itself. Predicting bottlenecks requires understanding the entire execution graph of the agentic task, just like for microservices, but that execution graph tends to be bigger and the tooling is less mature.

Handling Cognitive Failures Operationally

Microservice resilience focuses on surviving infrastructure failures using patterns like Retries, Timeouts, Circuit Breakers (configured in the application or service mesh; AWS App Mesh reaches end of support on September 30, 2026), Bulkheads, and Redundancy. Operationally, handling these involves monitoring infrastructure health, configuring resilience patterns appropriately, and having automated recovery mechanisms (like instance/container replacement via Auto Scaling). Debugging often involves analyzing logs and traces (using AWS X-Ray) to identify the failing network hop or service instance reporting an error.

Agentic AI must handle these infrastructure failures (especially for tool calls) but adds the significant operational challenge of dealing with cognitive failures: the AI producing incorrect, biased, unsafe, or nonsensical outputs (hallucinations) or failing to follow instructions or plan effectively. Detecting and debugging these is operationally different and a lot more difficult:

• Detection: While infrastructure failures throw clear error signals, cognitive failures might result in a successfully completed task with a subtly wrong or harmful outcome. Detecting this often requires implementing semantic validation checks on the agent's output (typically with Guardrails that can block the response generation), continuous monitoring against predefined quality metrics or 'golden datasets', or even incorporating human feedback loops. Simple health checks are insufficient, and LLMs don't reliably return accurate response codes (200 for OK, 500 for error, etc).

• Debugging: Imagine an agent consistently failing to extract the correct information using a tool, and imagine you've reliably identified this failure. Debugging a microservice failure might involve checking the tool's logs for errors. Debugging the agent failure, however, requires a multi-faceted investigation: examining the exact prompt sent to the LLM, the specific context provided (including any RAG results), the LLM's generated reasoning trace or intermediate 'thoughts' (if logged, and not all providers give you this), the parameters passed to the tool, the tool's actual response, and how the agent interpreted that response. This requires significantly richer logging and specialized debugging skills focused on the interplay between prompt, context, model behavior, and tool interaction. Standard distributed tracing often lacks the semantic depth required, and again we're faced with the lack of tool maturity.

• Mitigation Overhead: Implementing AI-specific resilience patterns like Reflection involves managing the logic for self-critique (e.g. have the agent review its plan against constraints or use a separate LLM call to evaluate its own response) and the retry mechanisms, adding latency and cost (more LLM calls). Setting up and monitoring human-in-the-loop workflows for exceptions adds significant process overhead. In the lab, these techniques work very well in most cases (no matter how much people like to criticize LLMs). In the real world, the level of work required, and the cost and especially latency introduced, often make them not really viable. Plus, you'll find a lot less people who know what Reflection is compared to the people who know what a Circuit Breaker is.

Observability and Tooling Maturity (May 2025)

I've already mentioned this a lot in the section above, but I think it's worth it to go beyond “the tooling is less mature”, and discuss what capabilities are missing. Just note that this whole field is so new that I forgive all the tool makers, even if I keep calling out their tools.

Seeing the "Why", not Just the What

The need for robust observability is amplified for agentic systems because of their non-determinism and the opacity of LLM reasoning. The standard three pillars of observability provide a baseline:

• Metrics (Amazon CloudWatch): Useful for tracking infrastructure health (Lambda, databases), tool invocation rates/errors, and critically, LLM usage metrics (token counts, inference latency, as provided by Bedrock or SageMaker).

• Logs (Amazon CloudWatch Logs, Amazon OpenSearch Service): Essential, but need to capture far more than typical application logs. Effective agent logs must include: the final prompt sent to the LLM (with RAG context), the model configuration used (model ID, temperature), the raw LLM response (including any intermediate reasoning or planned steps), details of tool calls (chosen tool, parameters, response), data retrieved from KBs, and the final output or action taken. Capturing this level of detail systematically is critical, but trust me, logs can get really noisy.

• Traces (AWS X-Ray): Useful for tracing requests across agent tool calls if those tools are instrumented microservices or Lambda functions. However, standard tracing tools generally cannot provide visibility inside the LLM's reasoning process or easily correlate distributed trace spans back to the specific semantic context or reasoning step within the agent that initiated them. I know for a fact that AWS is working on improving this, but we'll have to wait a bit more.

Observability 2.0 (which involves wide events as a single source of truth) is, to my knowledge, still absent in the Agentic AI space. I'll update this article if I hear something different.

The primary observability challenge for Agentic AI is achieving visibility into the "Why?" behind an agent's behavior. This requires specialized approaches, potentially integrating detailed logging with tracing tools or using specific features within agentic frameworks or platforms designed for visualizing execution graphs and reasoning steps. We can use standards like OpenTelemetry for LLM calls, but we're still lacking any standards specifically for representing and observing agent reasoning paths, making deep debugging and performance analysis harder than in the more standardized microservice world. Consequently, when planning an agentic system, teams must budget significant effort not just for development, but for building custom observability solutions or carefully evaluating the introspection capabilities offered by managed platforms.

Tooling Maturity and Platform Dependence

This observability gap reflects a broader difference in ecosystem maturity. The microservice world benefits from years of development and standardization efforts, particularly via the Cloud Native Computing Foundation (CNCF) centered around Kubernetes. This provides a rich ecosystem of mature, often interchangeable, open-source and commercial tools for orchestration (Kubernetes itself via Amazon EKS, or alternative orchestrators like Amazon ECS), service mesh (Istio or Linkerd), CI/CD, security scanning, monitoring, logging, and tracing. This standardization grants significant platform portability and flexibility, and more importantly, it turns microservice operations into a known and mostly solved problem (even if the solution is hard and expensive).

The operational tooling landscape for Agentic AI (MLOps/LLMOps) is much younger, more fragmented, and lacks comparable standardization. While excellent libraries exist for building agents, the tools for deploying, managing, observing, and securing complex multi-agent systems at scale are still rapidly evolving. There is no "Kubernetes for Agents" providing a universally accepted, platform-agnostic operational control plane.

This gap forces difficult operational choices:

• DIY Approach: Teams can build agentic systems using open-source frameworks and fundamental AWS services (Lambda, Step Functions, SQS, EventBridge, SageMaker for model hosting, OpenSearch for KBs). This offers maximum flexibility and avoids lock-in but requires significant engineering effort to build and maintain the custom orchestration, state management, observability, and deployment infrastructure reliably.

• Integrated Platform Approach: Leveraging platforms like Amazon Bedrock Agents provides pre-built, managed capabilities for agent creation, orchestration, tool integration, knowledge bases, and potentially simplified deployment and logging. This dramatically accelerates development but often results in significant platform dependence, tying the architecture and operational model to AWS-specific services and abstractions, potentially limiting future portability or requiring rework if migrating later. You're reading a newsletter called Simple AWS, so you know I'm cool with being locked in with AWS, but it's always important to call this out.

This gap in tooling was also one of my main drivers for writing this series. I was hoping to find lessons from microservices that we could apply to AI agents, but most things weren't that useful or easy to transfer to this domain.

Architects must therefore factor MLOps/LLMOps tooling maturity and the resulting platform dependence trade-offs into their risk assessment and strategic platform choices when designing agentic systems.

Deployment and Lifecycle: MLOps and LLMOps

MLOps and LLMOps aren't terms that most architects know about, but I'll give you a tip: ML engineers/architects who've been doing traditional ML since before ChatGPT know a lot of useful things about these topics; you should talk to them. I've been doing that a lot lately, and this section is a result of that.

Managing the AI Artifact Lifecycle

Microservice deployment typically follows mature Continuous Integration and Continuous Delivery (CI/CD) practices focused on code artifacts. Automated pipelines (using tools like the AWS CodePipeline, CodeBuild and CodeDeploy, or alternatives) handle building container images or function packages, running automated tests (unit, integration), deploying via strategies like Blue/Green or Canary, and monitoring the release.

Deploying and managing Agentic AI systems requires a slightly different set of practices called MLOps (Machine Learning Operations) and LLMOps, encompassing a more complex set of artifacts:

• Models: Versioning, deploying new or fine-tuned FMs, A/B testing performance, monitoring for drift or regressions.

• Prompts: Treating prompts as code: version control, automated testing (evaluating behavioral impact of prompt changes), safe rollout strategies. Prompt engineering is iterative and requires tight feedback loops.

• Knowledge Bases: Pipelines for updating KB data, re-indexing vector stores, validating data quality, and managing different KB versions.

• Tools: Standard CI/CD for the underlying code implementing agent tools.

• Agent Configuration/Orchestration: Versioning and deploying the definitions that tie models, prompts, tools, and workflows together. With Bedrock you can treat this as infrastructure.

• Evaluation: CI/CD pipelines must integrate rigorous automated evaluation suites that test the behavior and quality of agent responses against predefined benchmarks or criteria, going far beyond simple code compilation or unit tests for tools. I'll write more about this in another article.

This composite lifecycle, managing interconnected changes across code, models, data, and prompts, is inherently more complex and requires specialized MLOps/LLMOps tooling and processes that are still maturing. It's not that we don't know how to manage each of these parts individually. The problem is that the behavior of the system is determined by the combination of all of these parts, so we need to manage them as a single set, not independent variables.

Deployment Independence

The fact that these artifacts are so interconnected makes independent deployments more difficult. While a microservice team can confidently deploy their updated service if the API contract holds (that's one of the core points of microservices), updating an agent's prompt might subtly change its interaction with multiple tools or its interpretation of context, requiring broader behavioral testing. Swapping an underlying FM version often requires re-evaluating all associated prompts and tool interactions. This tighter coupling, especially within integrated platforms, makes the practical reality of independent deployment for agentic systems currently less attainable than the ideal achieved in well-architected microservice systems. Again, maturity.

AI vs Microservices Cost (May 2025)

Cost management is another area where the paradigms diverge significantly. In theory you could run your own models or open source ones in your own infrastructure, and the comparison becomes moot. In practice you'll often find yourself using proprietary models, and that's where you need to understand inference costs and tokenomics.

Contrasting Cost Models: GB-seconds and Tokens

Microservice costs on AWS are primarily driven by infrastructure consumption, scaling relatively predictably with usage: compute time (EC2, Fargate, Lambda), database capacity and I/O (RDS, DynamoDB), data storage (S3, EBS), network throughput (ELB, API Gateway, Data Transfer), and messaging volume (SQS, SNS, EventBridge). Costs can be tracked and optimized using standard cloud practices and tools like AWS Cost Explorer, and optimized with Savings Plans and Reserved Instances. If we used serverless compute, this is measured in GB-seconds (hence the title).

Agentic AI costs include these infrastructure components (for tools, KBs, orchestration compute) but are frequently dominated by a factor unique to AI: Foundation Model inference costs, typically priced per token. These are the main points you need to understand:

• Token Calculation: Costs apply to both input tokens (the prompt, including instructions, context, RAG results, chat history) and output tokens (the generated response, reasoning steps, or tool parameters). Longer interactions, more complex reasoning, or verbose outputs directly increase costs.

• Model Variance: Costs per token vary dramatically between different LLMs. Larger, state-of-the-art models like Claude 3.7 Sonnet can be orders of magnitude more expensive than smaller, faster models like Llama 4 Scout 17B.

• Task Complexity Impact: A simple agent query might use a few hundred tokens. A complex multi-step task involving extensive reasoning, RAG lookups, and multiple tool interactions could consume tens or even hundreds of thousands of tokens for a single user request. This makes cost highly sensitive to workflow design.

This token-based pricing makes agentic system costs potentially much more variable and harder to predict than typical microservice costs, which scale more directly with request volume or provisioned infrastructure. A small change in user input could trigger a much longer reasoning path, drastically increasing the token count and cost for that single interaction. Implementing robust cost monitoring, potentially with per-user or per-task tracking and alerts specifically for token consumption becomes more important than ever.

Optimization Strategies: Infrastructure Tuning vs. AI Efficiency

Microservice cost optimization primarily focuses on infrastructure efficiency: right-sizing compute instances and Lambda memory, using AWS Savings Plans or Reserved Instances for baseline load, leveraging cheaper Spot Instances where applicable, optimizing database performance, and implementing data caching with services like Amazon ElastiCache.

Agentic AI also requires these infrastructure optimizations for its non-AI components, but must also prioritize AI efficiency:

• Strategic Model Selection (Cascading): This is often the most impactful optimization. Use the simplest, cheapest model suitable for each step. Design workflows that route simple tasks to cheap models and only invoke powerful, expensive models when complex reasoning is truly needed.

• Prompt Engineering: Meticulously craft prompts to be concise, reducing input tokens. Guide the model to produce shorter, focused outputs where appropriate to reduce output tokens.

• Managing FM Costs: For predictable high volumes, evaluate Amazon Bedrock Provisioned Throughput to potentially lower per-token costs compared to on-demand usage, although this requires accurate capacity planning.

• Efficient Workflow Design: Minimize unnecessary LLM calls within an agent's workflow. Optimize the number of reasoning steps or tool interactions required.

• Effective Caching: Cache LLM responses (if determinism allows and inputs repeat), knowledge base lookups, or results from read-only tools when cache keys preserve access scope and expiry meets freshness requirements to avoid redundant computation and inference costs.

For agentic systems, tuning prompts and strategically selecting models often gets you greater cost savings than purely optimizing the underlying Lambda function memory. That's because in agentic systems and in any generative AI systems costs are typically dominated by inference. So, follow the usual advice: Start by optimizing the biggest thing you can find.

Part 3 Conclusion

So far in this series we've navigated the complex relationship between Microservice and Agentic AI architectures. We started by discussing their distinct origins and foundational principles, moved through the runtime dynamics regarding communication, state, and predictability, and now we've examined the practical realities of operating these systems.

While both patterns employ decomposition to manage complexity, they are fundamentally different tools designed for different primary purposes. Microservices offer a mature, robust paradigm for structuring large applications around business domains, optimizing for engineering lifecycle agility, operational scalability, and reliability through well-understood patterns and tooling. Agentic AI provides a powerful, rapidly evolving paradigm for automating complex tasks, enabling autonomous reasoning and action, and creating intelligent interactions by orchestrating LLMs, tools, and knowledge.

We've gone through all the comparisons I considered worth exploring. In the next part, part 4 of this series, we'll explore hybrid architectures and how we can apply both patterns at the same time. Stay tuned!

In Part 1 of this series I talked about the origins and foundational principles distinguishing Microservices from modern Agentic AI. I argued that microservices stem from software engineering needs, decomposing large applications along business domain lines to enhance agility and scalability. Agentic AI, fueled by Large Language Model (LLM) breakthroughs, focuses on decomposing complex tasks to orchestrate autonomous reasoning and action. Their different motivations shape their approaches to decomposition, specialization, and even the focus of autonomy itself (Team/Deployment vs. Operational/Decision-Making).

With that foundation laid, we now shift focus to how these architectural differences play out in runtime behavior. How do services communicate, versus how agents interact and collaborate? How do we manage the persistent records of a business versus the operational context of a reasoning engine? What patterns have emerged to solve common runtime problems on both sides, and how do they compare? And how does the fundamental nature of their core components impact system predictability? Exploring and contrasting these runtime dynamics is necessary to understand the practical trade-offs and design considerations inherent in each approach, which is what architecture is all about.

Interaction Styles of Microservices and Agentic AI

Both microservices and AI agents are distributed system, and in any distributed system the key is how those components communicate with each other. This isn't necessarily dictated by what the component do, but rather by the granularity of their responsibilities, and the expectations set upon the interactions.

Microservice Communication: Defined APIs and Simple Transport

Microservices interact primarily through well-defined Application Programming Interfaces (APIs). The common mechanisms include:

• Synchronous Communication: A service makes a request to another service's API and waits for a response. REST over HTTP/S is simple and stateless. For higher performance or scenarios requiring bidirectional streaming, gRPC (using HTTP/2 and Protocol Buffers) is a frequent choice, with the main alternative being websockets. This pattern is good for interactions where the caller requires immediate data or confirmation to proceed, though it has the disadvantage of introducing temporal coupling between the services, and can make some failure scenarios harder to recover from.

• Asynchronous Communication: This is a more complex pattern, but usually preferred in microservice architectures. Services communicate indirectly without waiting for an immediate response, typically via messaging systems. This promotes loose coupling and improves resilience, as components don't need to be available simultaneously. Common patterns and their associated AWS services include:

  • Message Queues (Amazon SQS): One service sends a message to a queue; another service polls the queue and processes the message later. Excellent for decoupling tasks and load leveling (throttling).

  • Publish/Subscribe (Amazon SNS): A service publishes a message (event) to a topic, and multiple subscribing services receive a copy independently. Ideal for broadcasting state changes or events, and for fanning out an event to multiple components. Can be used in conjunction with queues.

  • Event Buses (Amazon EventBridge): A central bus receives events from various sources (other microservices, or even AWS services) and routes them to target services based on defined rules. Facilitates building event-driven architectures.

  • Streaming Platforms (Amazon MSK for Apache Kafka, Amazon Kinesis Data Streams): For handling high-volume, ordered streams of events, often used for real-time data processing or feeding analytics pipelines.

A central philosophy guiding microservice communication is "smart endpoints, dumb pipes." The "smarts," the business logic, resides entirely within the microservices themselves (the endpoints). The communication infrastructure (network protocols and message brokers, the "pipes") should ideally act as simple, reliable transport mechanisms. This principle arose partly as a reaction against older SOA patterns where heavyweight Enterprise Service Buses (ESBs) sometimes contained significant business logic, transformation rules, or complex routing, creating a central point of complexity and potential failure. Intermediaries like an API Gateway (Amazon API Gateway) manage external API exposure, request routing, authentication, and rate limiting. Load Balancers (like AWS Elastic Load Balancer, both Application Load Balancers and Network Load Balancers) distribute traffic across service instances. Service Meshes (like Istio/Linkerd, and to some extent Amazon ECS Service Connect; AWS App Mesh reaches end of support on September 30, 2026) handle network-level concerns like reliable service-to-service communication, discovery, observability, retries, and circuit breaking. These intermediaries primarily provide infrastructure capabilities, aiming to keep application logic out of the communication fabric itself.

Agentic Communication: Context, Tools, and Orchestration

Agentic AI systems exhibit more diverse interaction patterns, reflecting the agent's need to reason, access information, and act upon the world:

• Agent-Tool Interaction: This is a fundamental part of agentic architectures. An agent, guided by its reasoning process (an LLM call), determines it needs to perform an action or get external information and invokes a predefined Tool. Technically, this is often implemented as a standard API call (REST or RPC) to a function (like an AWS Lambda function) or another service. The agent's LLM often needs to generate the correct parameters for the API call based on the task context and the tool's description, and that's a whole other discussion.

• Agent-Knowledge Base Interaction: Agents often need external information not present in their training data. Retrieval-Augmented Generation (RAG) is the common pattern. The agent (or a process acting on its behalf) takes the user query or intermediate thought, converts it to an embedding, searches a vector database (like those powered by Amazon OpenSearch Service with k-NN, or Amazon Aurora with the pgvector extension) for relevant text chunks, retrieves those chunks, and then includes them as context within the prompt sent to the LLM to generate a more informed response. This is a specialized information retrieval interaction.

• Agent-Agent Collaboration: In multi-agent systems, which are one form of agentic AI, coordination and ensuring the right agents are called is critical. This often involves patterns more complex than simple API calls:

  • Orchestrator/Supervisor: A central agent manages the workflow, assigns tasks to worker agents, gathers results, and decides the next steps, often communicating via internal function calls or messages.

  • Broker: A routing agent directs messages or tasks between specialized agents based on content or required skills, typically using asynchronous messaging. This function can also be performed by the Supervisor.

  • Event-Driven: Agents publish events reflecting their state changes or completed actions (e.g. to Amazon EventBridge), and other agents subscribe and react autonomously.

  • Frameworks like LangGraph allow defining these interactions as state machines or graphs, managed by the framework's runtime. Platforms like Amazon Bedrock Agents provide managed orchestration capabilities.

Critically, the communication payload in agentic architectures often includes rich context, intermediate reasoning steps, goals, or detailed instructions, not just simple data. The "dumb pipes" philosophy doesn't neatly apply. The agent's reasoning before calling other agents injects intelligence and business logic into the communication. Intermediaries like Orchestrators or frameworks explicitly contain workflow logic. The interaction is less about simple data transport and more about orchestrating an intelligent, context-aware process.

Managing State in Microservices and AI Agents

State is anything that is used for more than one request, and that should be persisted across requests and accessible to all nodes (a critical thing for high availability and scalability). This is true for both microservices and AI agents, but the way these architectures handle state and data persistence is another area of significant divergence.

Microservices: Decentralized Business Data & Eventual Consistency

The mantra here is decentralized data management. Each microservice owns its data, tailored to its needs (sometimes using different data stores like Amazon RDS for relational data, Amazon DynamoDB for NoSQL, etc.). This avoids the coupling and schema evolution nightmares of shared databases, and decouples other services that consume this data from how the data is stored. State primarily represents the authoritative, persistent records of business entities and transactions.

The major challenge this approach introduces is maintaining data consistency across these independent databases when a single business process modifies data owned by multiple services. Because distributed transactions using protocols like two-phase commit are complex and tend to make overall availability a bit more difficult, the prevalent approach is eventual consistency. This acknowledges that updates across services won't be instantaneous but will converge over time, requiring applications to tolerate temporary inconsistencies (e.g. an order status might briefly differ between the Order service and a Reporting service).

Handling this requires specific patterns:

• Saga Pattern: This pattern manages distributed business transactions through a sequence of local transactions within each participating service. If any step fails, predefined compensating transactions are executed to undo the effects of preceding successful steps, effectively rolling back the business operation. For example, if booking a flight succeeds but charging the credit card fails, the compensation action is to cancel the flight booking. Sagas can be implemented via:

  • Choreography: Services react to events published by others (e.g. using Amazon SNS or EventBridge). This is highly decoupled but can be hard to track.

  • Orchestration: A central coordinator (potentially implemented using AWS Step Functions) explicitly tells each service what local transaction to execute and handles failure/compensation logic. This is easier to manage but introduces a coordinator dependency.

• Transactional Outbox Pattern: This pattern ensures that events corresponding to a database change are published reliably. The service atomically commits both the data change and an event record describing the change to its local database within the same transaction. A separate process then reads these event records from the "outbox" table and reliably publishes them to a message broker (like Amazon SNS, EventBridge, or MSK). This prevents the state where the database is updated but the corresponding event notification fails to send.

Agentic AI: Memory, Knowledge, and Contextual Coherence

Agentic AI's focus is different. It's less about owning persistent business data and more about managing the operational state and context needed for reasoning and task execution. Key elements include:

• Memory: This is essential for maintaining context during interactions.

  • Short-Term Memory: Holds information relevant to the current conversation turn or task execution sequence (e.g. recent user messages, agent's intermediate thoughts or plans). This might be managed in memory or stored transiently. Context window limitations of LLMs are a major factor here, and even though they've grown a lot lately (Gemini 2.5 pro has a 1M tokens context window), you still need to be mindful of input tokens and the associated cost.

  • Long-Term Memory: Allows agents to recall information across multiple interactions or sessions (e.g. user preferences, past conversation summaries, learned facts). This requires persistence, potentially using databases like DynamoDB or, increasingly, vector databases for semantic retrieval of relevant past experiences.

• Knowledge Bases (for RAG): Agents frequently access external information repositories via Retrieval-Augmented Generation. The RAG process typically involves embedding a query, searching a vector database (OpenSearch or Amazon Aurora with pgvector) to find relevant document chunks, and injecting these chunks into the LLM prompt as context. This external knowledge isn't state the agent owns, but state it accesses to generate informed responses. Updating these knowledge bases isn't really part of the agentic architecture itself, but it's worth mentioning that they are updated.

The primary "consistency" challenge here is ensuring the agent's internal state (memory) remains coherent and relevant, and that the external knowledge bases are accurate and up-to-date. An agent reasoning with inconsistent memory or outdated knowledge will produce poor results. While multi-agent systems modifying shared state could face distributed consistency issues, patterns for managing inter-agent state consistency seem less standardized than microservice data patterns, likely because it's a much less prevalent pattern (at least according to my own experience, and likely because agents are so new). However, when an agent uses a tool that interacts with a traditional database or microservice, it must engage with that system's consistency mechanisms. For instance, if an agent orchestrates actions via tools that modify multiple backend microservices, implementing or invoking a Saga might be necessary to ensure the overall business operation completes reliably or compensates correctly.

Caching Strategies

Both architectures benefit from caching, and it's interesting to explore how they use it. Microservices use caches like Amazon ElastiCache (Redis, Memcached) primarily to store frequently accessed domain data or API responses, reducing latency and database load. Agentic AI caching focuses on reducing latency and cost (especially LLM token costs). Common targets include prompts (check out Amazon Bedrock Prompt Caching), LLM responses (if inputs are identical and some level of determinism is acceptable), knowledge base lookup results (which are the same idea as caching database responses), and outputs from read-only tool calls when cache keys preserve access scope and expiry meets freshness requirements (same idea as caching microservice responses). The performance and cost-saving goals are similar, but the nature of the cached content differs. Still, caching is caching, and there's a lot to learn from 15 years of microservice experience, and 50 or so years of cache experience (the earliest reference I could find is from Structured Computer Organization by Andrew S. Tanenbaum, 1976).

Shared and Differing Design Patterns

Both Microservice and Agentic AI architectures, being distributed systems, inevitably face similar fundamental challenges. However, the different nature of the components (deterministic code vs. probabilistic reasoning engines) and the core objectives (application structure vs. task automation) lead to divergent, though sometimes conceptually related, patterns. Let's explore how each architecture addresses a few shared problems, and consider whether they also share solutions.

Dynamic Component Discovery

In any dynamic distributed environment, particularly those leveraging cloud elasticity or container orchestration, component instances are ephemeral. Services or agents are created, destroyed, scaled up or down, and their network locations (like IP addresses and ports) are not fixed. This presents a fundamental problem: how does a component reliably find the correct network address of another component it needs to communicate with at runtime?

The microservice ecosystem relies heavily on Service Discovery mechanisms operating at the network infrastructure level. Typically, this involves a Service Registry (like AWS Cloud Map, Consul, etcd, or integrated Kubernetes service discovery). When a microservice instance starts, it registers its network location and health status with the registry. When another service needs to call it, it queries the registry using a stable logical service name to resolve the current, healthy IP address(es) and port(s). Load balancers often integrate directly with these registries to know where to distribute traffic. The focus is squarely on resolving logical service names to physical network endpoints.

Agentic systems also need discovery, but often at a different level of abstraction. While an agent might need to discover the network endpoint of a Tool (which could use standard service discovery), a more common challenge is “discovering” the right tool or agent for a specific task. This frequently involves the core LLM's reasoning capabilities. The agent is provided with descriptions of available tools (including their purpose and parameters). Based on its current goal or sub-task, the LLM analyzes these descriptions to semantically determine the most appropriate tool to invoke. Agentic Frameworks might also provide registries or mechanisms for tool/agent lookup, but the selection process often involves this layer of semantic understanding or functional matching rather than just resolving a network address. In multi-agent systems, finding another agent might involve explicit routing by an Orchestrator or Broker, or discovery based on capabilities advertised within the framework.

Both architectures solve the "finding the right component" problem, but operate at different conceptual levels. Microservices know which component they're looking for, and they primarily rely on network-level service discovery to resolve logical names to physical addresses for potentially identical service instances. AI Agents first need to determine which component they're looking for, relying on semantic reasoning or framework-level capabilities to discover the functionally appropriate tool or specialized agent needed for a specific step in a task, and only after that they fall into the same problem of microservice discovery.

Managing Interaction Complexity & Routing

As the number of independent components (services or agents) grows, direct peer-to-peer communication becomes increasingly complex to manage, test, and secure. Systems need ways to route requests efficiently, manage external access, and control internal interaction flows without creating a tangled mess.

In a microservices architecture this is addressed through patterns that manage traffic flow and centralize certain concerns, primarily at the network edge or infrastructure level. The API Gateway pattern provides a single, managed entry point for external clients, handling concerns like request routing to appropriate backend services, authentication/authorization, rate limiting, request and/or response transformation, and sometimes response aggregation. Internally, Load Balancers distribute traffic across available instances based on load and health. Service Meshes can provide more sophisticated internal traffic management, like fine-grained routing rules or traffic splitting for canary releases. These solutions focus on managing network traffic and enforcing policies at architectural boundaries.

Managing interaction complexity in Agentic AI architectures often involves embedding logic within specialized agents or frameworks that control the workflow. The Orchestrator/Supervisor agent centralizes the control logic for a multi-step task or multi-agent collaboration. The Broker agent (which may be a separate agent, or may be the same Supervisor agent) intelligently routes messages or tasks to other agents based on content or required skills. Routing Workflow patterns explicitly define paths for different types of tasks to reach specialized handlers or agents. These patterns manage the logical flow of execution and collaboration, often making decisions based on the state of the task or the content of messages.

Both paradigms use intermediary patterns to manage interaction complexity. However, microservice intermediaries (API Gateway, LB) primarily function as infrastructure components, routing network traffic and enforcing generic policies without deep application awareness. Agentic intermediaries (Orchestrator, Broker) often encapsulate significant application or workflow logic, actively directing the sequence of operations and making content-aware routing decisions. The intelligence sits closer to, or within, the interaction management layer in agentic systems.

Ensuring Reliability & Handling Failure

Partial failures are a fact of life in distributed systems. Network connections can drop, components can crash or become slow, external dependencies can fail. Systems must be designed to tolerate these faults and continue operating reliably, or at least fail gracefully.

In microservices, reliability focuses heavily on infrastructure resilience and handling failures between deterministic code units. Key patterns include: Redundancy (running multiple instances), Health Checks (detecting unhealthy instances for removal from load balancing), Timeouts (preventing indefinite waits on dependencies), Retries (handling transient network errors), Circuit Breakers (stopping requests to failing services to prevent cascading failures and allow recovery), and Bulkheads (isolating resources to limit the blast radius of dependency failures). Asynchronous communication also contributes significantly to resilience by decoupling components. Data consistency (or at least detection of inconsistencies) is also an important aspect, but I already talked about it. The key point is that these patterns primarily address component unavailability or network issues.

Reliability in Agentic AI is twofold. First, it must handle the same infrastructure failures as microservices, especially when Tools interact with external systems (requiring robust tool design and standard patterns like Retries and Timeouts applied either by the agent logic or within the tool). Second, and uniquely, it must handle AI-specific or cognitive failures: LLM hallucinations, poor reasoning or planning, inability to use a tool correctly, misinterpreting results. Solutions for this include AI-specific patterns like Reflection/Self-Correction (where an agent reviews and attempts to fix its own output or plan based on predefined criteria or checks), Evaluations (external validation of agent performance, either by code or by another LLM call), adaptive Re-planning (finding alternative paths if a step fails), and robust Error Handling within the agent's core logic to catch and manage exceptions from both tools and the LLM. Guardrails act as safety nets against undesirable outputs, and they can be applied either to the final response or to intermediate responses.

The Predictability Divide: Deterministic Logic vs. Probabilistic Reasoning

This difference perhaps most profoundly shapes the operational reality and design philosophy of Generative AI applications in general, not limited to Agents.

Microservices, like most traditional software, are built upon a foundation of determinism. Given a specific input and state, a correctly functioning service is expected to produce the exact same output and undergo the exact same state transition every single time. This predictability is fundamental to how we:

• Test: Unit tests, integration tests, and end-to-end tests largely rely on asserting that actual outputs match expected outputs for given inputs.

• Debug: Reproducing a bug typically involves recreating the specific inputs and state conditions that reliably trigger the erroneous behavior.

• Reason about Reliability: We build confidence by testing predictable components and using patterns (like retries for transient network errors) that manage failures within an otherwise deterministic system.

It's worth noting that distributed systems are never truly 100% deterministic. Unexpected network and infrastructure failures can and will occur, and they will impact the system. These are mostly solved problems, even if the solutions are rather complex (distributed transactions, eventual consistency, etc). Moreover, we can test for these unpredictable failures using chaos engineering, for example with AWS Fault Injection Simulator. In almost any form of traditional software we adapt the application layer to handle these failures and test for them at the infrastructure layer.

Agentic AI: Embracing (and Managing) Non-Determinism

Agentic systems leveraging LLMs introduce inherent non-determinism. This stems primarily from the probabilistic nature of how LLMs generate text: they predict the most likely next token (or word part) based on the preceding context, often employing sampling strategies (like adjusting the 'temperature' parameter) to introduce variability and avoid repetitive outputs. This means:

• The same prompt might yield slightly different phrasing, sentence structure, or even minor variations in the reasoning path on subsequent runs.

• In rare cases, it can lead to hallucinations: confident-sounding but factually incorrect or nonsensical outputs.

This non-determinism is not just unavoidable, it's often entirely desirable. Any form of creativity by an LLM is due to the same factors that cause hallucinations, and the whole reason why we build Generative AI applications and use LLMs is because of these factors. However, even if we're explicitly searching for this non-determinism, its presence fundamentally changes how we operate applications:

• Testing: Exact output matching is often impossible or irrelevant. Testing must shift focus to behavioral validation: Does the agent achieve the intended goal? Does the output satisfy key criteria (checked via rules, another LLM, or human evaluation)? Is the agent robust to slight variations in input phrasing? Testing often requires evaluating performance over multiple runs or using statistical measures, and determining correctness via semantic analysis instead of direct comparison. Equivalence classes for inputs are often impossible to predict just by analysis, and likely impractical even to calculate via statistical observations. For these reasons, significant investment in robust evaluation frameworks becomes necessary, and I promise to write an article about it soon.

• Debugging: When an agent misbehaves, pinpointing the cause is harder. Was it the prompt, the specific context window contents, retrieved RAG data, the inherent randomness of the model for that query, or a genuine hallucination? Reproducing the error can be difficult, even if you manage to recreate the exact state. Debugging necessitates capturing and analyzing the entire reasoning context: the exact prompt sent to the LLM, the retrieved knowledge snippets, any tool calls and responses, and the intermediate "thoughts" or plans generated by the agent (requiring detailed logging or specialized observability tools). To make matters even more fun, there are no tools that can really observe the internal state of a model, so we can only capture what the model outputs.

• Reliability: Building reliable agentic systems means managing this uncertainty. It requires good input validation, strong output validation (checking against rules or desired formats, and very robust evals), incorporating self-correction mechanisms (like the Reflection pattern), implementing clear fallback behaviors when high confidence cannot be achieved, and using Guardrails to prevent harmful or undesirable actions. Human oversight (called human-in-the-loop) for critical tasks is often a necessary reliability component, even if introduces delays and causes more work.

Conclusion: Runtime Realities Shape Design

The runtime behaviors of Microservices and Agentic AI systems present some similarities, but also differences due to the fundamental nature of their components. Microservices communicate via relatively simple transport mechanisms between endpoints containing deterministic business logic, manage persistent business data with established patterns for eventual consistency, and rely on infrastructure-focused resilience patterns. Agentic AI involves more complex, context-rich communication flows often managed by intelligent intermediaries, focuses state management on operational context and memory, requires unique patterns for cognitive reliability, and must contend with inherent non-determinism resulting from its AI core.

In part 1 we concluded the different focus (especially the why) of these two architecture patterns results in different ways to solve the same problems, due to their different priorities. Here in part 2 we found that some solutions apply equally well to both patterns, but in many cases AI Agents add one more layer to the problem, necessitating newer solutions on top of the known patterns. In part 3 we'll analyze deployment, observability and security, and consider hybrid implementations that combine both patterns.

Modern system design is about dealing with complexity. As applications grow, monolithic approaches can become bottlenecks, affecting development speed, scalability, and resilience. That's where we start applying architectural patterns focused on decomposition, like Microservices (though remember it's not the only option).

When you start talking about AI Applications, especially related to Generative AI, at some point you face the same problem: That big ball of mud becomes really hard to maintain, and especially to operate. Agentic AI seemingly comes to solve that problem, decomposing your AI application into individual agents.

Both patterns involve breaking down larger systems into smaller, interacting units, and both find strong support within cloud ecosystems like AWS. So it makes sense to think they're similar, or even the same thing applied to different contexts. That assumption is what prompted me to dive into this comparison, hoping to find lessons from microservices architecture that I could apply to Agentic AI.

But I was wrong.

Looking past these surface similarities reveals fundamental differences in the origins, the core purpose, and the design philosophies of these patterns. And while there are shared lessons, the main lesson is that they are not the same.

This article is the first in a series dedicated to a deep comparison of these two architectural patterns. I'll explore the underlying principles and practical consequences of each, in an attempt to build a clear understanding of the distinct problems each paradigm solves best and the unique ways they approach system design. Here in Part 1, I establish the foundation, examining their contrasting histories and motivations, their vastly different approaches to decomposition and specialization, and the distinct meanings they assign to autonomy and independence. Getting these fundamentals right is necessary before we can effectively analyze their runtime behaviors and operational realities in subsequent parts.

Introduction: Beyond Surface Similarities

The appeal of both Microservices and Agentic AI lies in their promise to manage complexity through decomposition. Microservices carve up large, monolithic applications into collections of smaller, independent services. Agentic AI breaks down complex tasks or goals into sequences of reasoning and action executed by potentially multiple intelligent agents. Both yield distributed systems operating on cloud infrastructure.

And both have become very popular, in a memetic and fashion-like way that's unrelated to their actual usefulness. They're the cool kids on the bloc, and if you use them you're cool and knowledgeable and your seniority magically increases.

And that's where the resemblance ends. The reasons driving this decomposition, the nature of the resulting components, and the fundamental challenges being addressed diverge significantly. Microservices emerged as a software architecture pattern focused squarely on improving the software development lifecycle – enhancing team agility, enabling independent deployment, achieving better scalability, and increasing the resilience of large applications. It's about structuring code and teams more effectively. Agentic AI, especially its current LLM-driven form, is an AI system design pattern focused on achieving sophisticated, goal-oriented, autonomous behavior. It aims to automate complex processes involving reasoning, planning, learning, and interaction with the environment. It's about orchestrating intelligence.

They both divide complexity into more manageable pieces, but they do it for different reasons, and they divide different complexities.

Misunderstanding this core distinction, or trying to apply principles from one directly to the other without careful consideration, can lead down unproductive architectural paths. Let's begin by examining their roots so we can understand the why.

Different Problems, Different Origins

An architecture pattern's history often reveals its soul. The core problems it was conceived to solve heavily influence its structure and principles. The why determines the how.

Microservices: Basically REST-based SOA to fight Monoliths

Microservices represent an “evolutionary refinement” (I'm feeling fancy 🤣) of earlier distributed system ideas, particularly Service-Oriented Architecture (SOA). SOA became prominent in the early 2000s, aiming for better enterprise integration and component reuse via standardized services that communicated via the SOAP protocol over a centralized Enterprise Service Bus (ESB). It had a pretty significant impact in enterprise architecture, but certain SOA implementations grew way too complex and sometimes introduced performance or governance bottlenecks.

Around the early 2010s, practices honed at web-scale companies like Netflix and Amazon facing the extreme limitations of monolithic applications began to coalesce into the Microservices pattern. This approach embraced SOA's service orientation but adopted a more pragmatic, decentralized philosophy. It emphasized simpler communication styles (often via REST/HTTP), favored direct service-to-service interaction or lightweight messaging ("dumb pipes") over heavy middleware, advocated for decentralized data management (each service owns its data), and placed a very high premium on automation (CI/CD) and independent deployability.

The short explanation of microservices is that you decompose a use case into separate behaviors that use separate data, and implement the management of those units of data and the execution of those behaviors into a single deployment unit with a well-defined interface, called a microservice. The entire application is the result of combining the behaviors of these microservices in certain ways.

The motivation was explicitly to combat the pains of the monolith: development bottlenecks slowing down releases, inefficient scaling of the entire application for localized load, technology lock-in slowing down innovation, poor fault isolation leading to cascading failures, and difficulties coordinating large teams on a single codebase. Microservices were a direct answer to critical software engineering lifecycle and operational scaling challenges.

It's worth noting at this point that we have learned two critical lessons since: One, that we can build scalable monoliths with good code practices focused on separation of concerns. Two, we can build distributed services which are separated at the application layer but share the data layer, and while they don't offer all the advantages of microservices the get us most of them and are much easier to implement and maintain. Yes, in case you didn't know, if your “microservices” all access the same database they're not really microservices, and that's ok!

Also, modern cloud technologies have made monoliths, distributed services and microservices much easier to build, deploy, and especially operate.

Agentic AI: Managing LLMs for Complex Tasks

The current surge in Agentic AI architectures is much more recent and has a different catalyst: the dramatic arrival of powerful Large Language Models (LLMs) in the early 2020s, and their limitations. Early models like GPT-4 demonstrated impressive capabilities not just in language generation, but also in rudimentary reasoning, instruction following, and even planning. And on top of their real use cases, they also became a fad, greatly increasing their adoption.

However, it quickly became apparent that complex, real-world tasks often require more than a single LLM interaction. Tasks might involve multiple steps, require access to up-to-the-minute external information, necessitate interaction with other software systems or APIs, or need context maintained over extended interactions. This gap between LLM potential and practical task complexity spurred the use of multiple techniques and improvements like Retrieval-Augmented Generation (RAG), multi-turn conversations, and hybrid architectures combining regular code with LLM calls to solve a problem. This ultimately led to the development of agentic frameworks and architectures, where the core idea became to orchestrate LLM calls, augmenting the model's reasoning capabilities with external Tools (functions or APIs the agent can call), Memory (short-term context and long-term information retention), and access to external Knowledge Bases (often via RAG, using vector databases like Amazon OpenSearch or Amazon Aurora for Postgres with pgvector).

This modern wave actually builds upon decades of theoretical work in AI on autonomous agents (Distributed AI, Multi-Agent Systems, Belief-Desire-Intention models), but this time with LLMs as readily available reasoning engines, often orchestrated by platforms like Amazon Bedrock, or open-source frameworks. The primary motivation is extending AI capabilities to automate complex, multi-step tasks requiring planning, reasoning, and interaction, thereby achieving a higher degree of autonomous, goal-oriented behavior.

The Decomposition Divide: Structuring Applications vs. Structuring Tasks

As you can see, it's the same idea of decomposing a problem and creating autonomous units that can solve individual parts of it, but with a completely different reason why we're decomposing the problem in the first place. Why does the reason matter? Because it will determine how we structure the components and what we optimize the solution for.

Microservice Decomposition: Business Capabilities and DDD

Microservices decompose the structure of a software application. The goal is to partition the application into independent units that align with business functions, promoting agility, scalability and maintainability via independent components that can be owned separately. This decomposition is often guided by principles from Domain-Driven Design (DDD):

• By Business Capability: Services map directly to what the business does (e.g., Catalog Management, Pricing Engine, Shipping Logistics). Each service contains all logic and data needed for its capability. This ensures technology serves business needs directly, and makes owning the data easier.

• By Subdomain (DDD): The overall business domain is analyzed to identify distinct subdomains (e.g., Product, Order, Customer). Each subdomain with its specific language and model becomes a Bounded Context. Microservices are often designed to align with these Bounded Contexts, ensuring conceptual integrity and clear boundaries. Defining these boundaries often involves identifying Aggregates (entities treated as a unit) within the domain.

• By Transaction: Occasionally, services might be designed around encapsulating entire high-level business transactions. The idea of these functional microservices is to abstract away the implementation details of a distinct part of a process, along with the ownership of the data involved.

The guiding principles are High Cohesion within a service (related things stay together) and Loose Coupling between services (dependencies are minimized and managed through clearly defined APIs). Getting these boundaries right, often based on domain analysis, is essential for realizing the benefits. This often involves understanding the stable structures within the business domain itself, and understanding the data that is stored, and which processes use it and how.

Agentic AI Decomposition: Functional Roles and Workflows

Agentic AI decomposes a complex task or goal into manageable steps or functions. The goal is to create an executable plan or workflow that leverages the reasoning capabilities of LLMs and the specific functionalities of tools or specialized agents. The decomposition is driven by the logic of the task itself:

• By Functional Steps: Breaking down the task sequentially (e.g., Step 1: Understand request, Step 2: Search knowledge base, Step 3: Call Tool A, Step 4: Synthesize result, Step 5: Format output).

• By Role/Skill: Assigning different agents specific roles based on required skills or tools (e.g., a Planner Agent creates the task breakdown, Research Agents use specific search tools, an Executor Agent calls action APIs, an Evaluator Agent checks the quality). This mimics human team specialization.

• By Workflow Pattern: Implementing structures like a linear Pipeline, a centrally managed Orchestrator directing workers, concurrent execution via Parallelization, or dynamic message routing via a Broker. Frameworks like LangGraph allow defining these complex execution flows.

Here, the focus is on orchestrating the process of reasoning and action needed to achieve the goal.

Comparing Philosophies: Domain Stability vs. Task Logic

Microservice decomposition anchors itself in the relatively stable concepts of the business domain. The architecture reflects the business structure, aiming for long-term maintainability. Agentic AI decomposition anchors itself in the logic required to complete a specific task or goal. This logic might be more dynamic, changing as the understanding of the task evolves, new tools become available, or the capabilities of the underlying LLMs change.

Trying to apply DDD Bounded Context thinking rigidly to every step of an agent's workflow is likely inappropriate. While high-level agents might correspond to broad business domains, the fine-grained decomposition within an agent's process is typically functional or task-oriented. Understanding this difference is fundamental because choosing the wrong decomposition strategy can lead to poorly defined boundaries, increased coupling (either between services or between agent steps), and ultimately defeat the purpose of decomposition in either context. For microservices, it leads to systems hard to change independently. For agents, it can lead to brittle or inefficient task execution.

The functional microservices style of decomposition is the one that looks the most similar to Agentic AI. At a glance, the overall goal of a functional microservice is to abstract behind an API a set of steps or a transaction within an entire workflow, and workflows are realized by combining multiple functional microservices. Taken at face value, that's pretty much how you decompose a task into AI agents, and I believe now that was one of the core reasons for my initial idea that microservices and AI agents were very similar. Looking a bit deeper into it, a functional microservice also owns all the data that is involved in the transaction, and manages access to it. It encapsulates not just the behavior of executing the transaction, but also the transaction itself as an object of the system. AI agents operate at a much finer-grained level, and it would seem logical for a single functional microservice to implement its behavior using a fleet of AI agents.

Specialization: Domain Expertise vs. Functional Skill

The key point here is that while both architectures aim for specialized components, they specialize along different axes.

Microservice Specialization: Owning a Business Domain Slice

A microservice specializes in a specific segment of the business domain. The AuthenticationService knows everything about authentication protocols, credential storage, and session management. The RecommendationService understands recommendation algorithms, owns user preference modeling, and stores item metadata. This domain-oriented specialization allows teams to build deep expertise and optimize data models and logic for that specific business capability.

Agent Specialization: Specific Tasks and Tools

An AI agent typically specializes in a specific function, task, role, or skill. For example:

• An agent might be highly optimized for natural language conversation and managing dialogue state.

• Another might specialize in interpreting and executing code via a specific tool.

• A third might be expert at RAG, efficiently querying multiple vector databases and synthesizing retrieved information.

• Yet another could be a ValidatorAgent specialized in checking the outputs of other agents against specific rules or constraints.

This task/role/skill-oriented specialization aims to improve the agent's performance (accuracy, latency, reliability) and cost for its specific function, often by tailoring its prompts, carefully selecting and fine-tuning its underlying model (if applicable), or giving it access to a specific, curated set of tools and knowledge.

Implications for Design and Boundaries

This difference reinforces the boundary definition contrast. Microservice boundaries encapsulate domain knowledge. Agent boundaries encapsulate specific skills or steps within a process. An agent specialized in using the WeatherAPI tool has a boundary defined by that functional capability, which is conceptually different from a UserProfileService boundary defined by the scope of user-related business data and logic.

Autonomy and Independence: Empowering Teams vs. Empowering Decisions

Autonomy and independence are advertised as benefits of both approaches, but the emphasis and practical realization differ significantly.

Microservice Autonomy: Independent Deployment and Governance

Microservice autonomy is primarily about empowering development teams and enabling operational independence for deployable units:

• Team Autonomy & Decentralized Governance: Teams owning services can often choose their own technology stacks (within organizational bounds) and manage their service's evolution.

• Independent Deployability: This is the cornerstone. The ability to deploy a single service update without impacting others, enabled by well-defined APIs and robust automation (like CI/CD pipelines running on AWS CodePipeline), is a massive driver of agility.

The service runs independently, but the key autonomy is about the team's control over its development and release lifecycle.

Agentic Autonomy: Task Execution Details

An AI agent perceives its environment (user input, tool output), reasons based on its goals and internal state (memory, model knowledge), and autonomously chooses and executes actions (calling tools, generating responses) to achieve its objectives. This is runtime, decision-making autonomy.

An AI agent also serves to encapsulate the execution details of the given task, centralizing management of the prompt, model, knowledge bases and tools used to execute that specific task.

Independence Realities: Encapsulation, Deployment Limits, and Ownership

• Encapsulation: Both architectures heavily rely on encapsulation through interfaces (APIs for microservices; agent interactions for agents). This is a strong parallel and a shared best practice that enables modularity. However, the granularity of the behaviors that each architecture encapsulates are different.

• Deployment Independence: This deployment comparison reflects the tooling landscape in April 2025. Microservices, especially when built using containers and orchestrated by platforms like Amazon ECS or Kubernetes on Amazon EKS, achieve a high degree of practical independent deployment across various environments. Agentic AI systems are conceptually modular, but in practice there is no standardized, platform-agnostic operational tooling for complex agent lifecycles. This heavily contrasts with the "deploy anywhere" flexibility that containerized microservices have. This platform dependence is a significant practical constraint, even on the more mature platforms like Amazon Bedrock.

• Team Ownership: The "you build it, you run it" model provides clear ownership for microservices, and as mentioned above, existing tools provide excellent support for this. For agentic systems, ownership is often ambiguous. Is it the prompt engineering team, the machine learning team, the tool-building team, or a central AI platform team? Clear ownership models are still evolving, complicated by the diverse skillsets required (AI/ML, Prompting, Software Engineering, Data Engineering).

Conclusion: Laying the Foundation (Preview of Part 2)

This initial exploration reveals that Microservices and Agentic AI, despite sharing the concept of decomposition, are built on different foundations to solve different core problems. Microservices refine software architecture principles to tackle application complexity and improve the development lifecycle, anchoring themselves in stable business domains. Agentic AI tackles complex tasks through autonomous behavior, anchoring itself in reasoning processes, functional roles, and workflow orchestration. Their distinct motivations dictate different decomposition strategies, specialization types, and degrees of autonomy and independence.

Ultimately, we need to remember why we're using an architecture pattern: You use microservices to achieve independent ownership and deployment of functionality around business domains. You use Agentic AI to implement specific complex behaviors that resolve specialized tasks in complex processes.

And more importantly, we need to remember that this is not an A or B decision. You can use both of these patterns at the same time, or you can use neither.

In Part 2 we'll build upon this foundation, examining how these different architecture patterns result in different communication techniques, state management approaches, design patterns, and the critical dimension of predictability at runtime.

I spend a lot of time working with large language models in production, and one persistent problem is that they're expensive 💸 . Prompt caching can help a lot with that, especially when you're reusing the context in a conversation (something that we take for granted in ChatGPT or Claude.ai, but is actually expensive).

This March 2025 article explores Amazon Bedrock's explicit prompt caching, checkpoint mechanics, cache lifetime, APIs, and economics. Model support, cache lifetimes, and prices have since changed; the historical code snippets require schema and response-handling repairs before use.

Technical Architecture of Bedrock Prompt Caching

Prompt caching isn't just a simple text storage mechanism. There's a whole infrastructure dedicated to preserving neural network states.

When a large language model processes text, it's not just reading characters, it's building an internal representation of everything it reads. This includes attention patterns, token relationships, and other neural activations. Normally, when you send a prompt, the model rebuilds this entire representation from scratch every time, even if 90% of the prompt is identical to previous requests.

What actually happens with Bedrock prompt caching is that the service captures this internal neural state at specific points (checkpoints) and stores it in an ephemeral cache. This is fundamentally more complex than just storing the text string, it's preserving the actual computed representation inside the model.

The cache itself lives in AWS-managed infrastructure, completely isolated within service boundaries to maintain security between different AWS accounts. Your cached content isn't accessible to other users, and their cached content isn't accessible to you. The architecture maintains security by keeping the cached state tied to your specific Bedrock resources and API credentials.

The caching layer integrates directly with Bedrock's execution environment. If you're familiar with how AWS Lambda works, you can think of a similar execution model. Just as Lambda runs on AWS Lambda Workers (which are essentially EC2 instances), Bedrock also runs on compute infrastructure optimized for inference. The prompt caching system sits between your API calls and this infrastructure, intercepting and storing neural states when instructed.

From a performance standpoint, setting up a cache creates a small overhead on first write. But that investment pays off dramatically on subsequent reads, with massive reductions in processing time and cost. The whole system is optimized for bursts of activity with similar prompt content, which fits the pattern of many LLM applications like conversation agents, document analysis tools, coding assistants, or anything where you keep asking things about something you already shared.

Cache Checkpoint Mechanics

Cache checkpoints are the actual mechanism that makes prompt caching work. But what exactly is a checkpoint? It's a specific position in a prompt where Bedrock saves the model's entire internal state up to that point. Think of it as a bookmark in the model's thought process, the model can jump back to that exact state later without reprocessing everything that came before.

You can't just place checkpoints anywhere you want. They follow specific token thresholds that vary by model. For Anthropic's Claude 3.5 model, you need approximately 1,024 tokens of combined conversation before you can set the first checkpoint. Why this minimum threshold? Because the overhead of caching very small prompts would outweigh the benefits, there wouldn't be enough tokens saved to justify the cache management cost.

Different models have different checkpoint configurations:

• Claude 3.5 allows up to 4 checkpoints per conversation

• Amazon's smaller Nova models typically support just 1 checkpoint

• The placement options vary by model, some allow checkpoints in system messages, user messages, and tool sections, while others restrict placement

The checkpoint creation process is pretty fascinating if you understand what's happening during model inference, which is why I'm diving so deep into this stuff (I had a lot of fun researching it!). When Bedrock identifies a cache point in your prompt that meets the token threshold, it does something akin to taking a snapshot of the entire neural network's state at that moment. This state captures all the attention patterns, embeddings, and internal representations that the model has built up to that point. This complex state is extracted and stored with a unique identifier associated with your AWS account and the specific model.

I should emphasize one technical limitation: checkpoints are tied to the exact prefix they follow. If you change even a single character before the checkpoint, the cache can't be used. Why? The model's internal state depends on the precise sequence of tokens it has processed, any change, no matter how small, would result in a different neural state. This strictness is necessary because the model's computations (not output) are deterministic based on the input.

Another important detail is how checkpoints interact with the model's context window. Cached content absolutely still counts toward the total context window for the model. If you cache a 2,000-token document and the model has a 100,000-token context window, you'll still have only 98,000 tokens available for additional content before the model starts “forgetting” stuff and quality goes to the floor. The cache doesn't magically expand the context window, it just prevents redundant processing of the same tokens, and prevents you from having to pay for that redundant processing.

Cache Lifetime and Management

In Bedrock's implementation, the prompt cache is ephemeral by design, with a default Time To Live (TTL) of 5 minutes. The TTL timer works on a sliding window basis, each time the cache is successfully hit (used), the 5-minute timer resets. This keeps frequently used cache entries alive while allowing rarely used ones to expire naturally. Why 5 minutes specifically? It's likely the AWS team found this to be the sweet spot that covers most interactive use cases without wasting storage on infrequently accessed data. I'll ask them if I get the chance.

When a cache expires, the stored model state is discarded completely. There's no way to retrieve it after expiration, and no background persistence occurs. If you need the same content again after expiration, you'll need to reprocess the full prompt and recreate the cache from scratch.

What's particularly interesting about Amazon's implementation is that it's clearly optimized for interactive workloads rather than long-running processes. The 5-minute window aligns with typical human interaction patterns in conversation systems and document analysis workflows, where bursts of activity occur within short timeframes. For systems with longer idle periods between related requests, you'll need to implement strategies to manage cache retention, such as periodic "keepalive" requests or prompt restructuring to minimize the impact of cache misses. Yes, both of these are real things we do in prod.

Bedrock API Integration

Understanding how to integrate prompt caching with Bedrock's APIs is where the rubber meets the road. There are three primary integration points: the Converse API for multi-turn conversations, the InvokeModel API for single-turn completions, and automatic integration with Bedrock Agents.

For the Converse API, which handles chat-style interactions, you mark cache points within the message structure. Here's a Python example:

import boto3

bedrock = boto3.client("bedrock-runtime")
document_text = "Very long reference document that you don't want to process repeatedly..." 

messages = [{"role": "user", "content": []}]
# Add the long context to the user message
messages[0]["content"].append({"text": document_text})
# Mark cache checkpoint after the large context
messages[0]["content"].append({"cachePoint": {"type": "default"}})
# Add the user's actual question after the checkpoint
messages[0]["content"].append({"text": "What are the key points in this document?"})

response = bedrock.converse(
    modelId="anthropic.claude-3-5-sonnet-20241022-v2:0",
    messages=messages
)

This structure tells Bedrock to process the document_text once, cache the resulting model state, and then process the question. On subsequent requests, resend the same cacheable document prefix and cache checkpoint; Bedrock can reuse its cached computation. The beauty of this approach is that it integrates directly with the conversation structure without requiring special caching APIs.

For the InvokeModel API, which handles single-turn prompts, caching is enabled by including appropriate fields in the request body. The exact structure depends on the model being used. For Anthropic models, it typically involves adding a cache_control field:

response = bedrock.invoke_model(
    modelId="anthropic.claude-3-5-sonnet-20241022-v2:0",
    body=json.dumps({
        "prompt": "System: You are a helpful assistant.\n\nHuman: " + document_text,
        "cache_control": {"type": "ephemeral"},
        "max_tokens": 1000
    })
)

Here's a trick that's not immediately obvious: To verify that caching is working properly, you should examine the response metadata. Bedrock includes cache-related metrics in the usage data:

# Extract usage metrics from response
usage = response["usage"]
cached_read_tokens = usage.get("cacheReadInputTokens", 0)
cached_write_tokens = usage.get("cacheWriteInputTokens", 0)

print(f"Tokens read from cache: {cached_read_tokens}")
print(f"Tokens written to cache: {cached_write_tokens}")

On the first request you'll see a high value for cacheWriteInputTokens (the tokens being cached) and zero for cacheReadInputTokens. On subsequent requests that use the same cached content, you'll see the opposite pattern, high cacheReadInputTokens and zero cacheWriteInputTokens. This provides a clear signal that the caching system is working as expected.

For Bedrock Agents, enabling prompt caching is even simpler, you just toggle a setting in the agent configuration:

response = bedrock_agent.update_agent(
    agentId="your-agent-id",
    promptOverrideConfiguration={
        "promptCachingEnabled": True
    }
)

With this setting enabled, the agent automatically manages cache checkpoints without requiring any additional code in your application logic. This is particularly useful for complex agent workflows where manually placing checkpoints would be cumbersome.

How Much Did Bedrock Prompt Caching Cost in March 2025?

Now let's talk about money. The economics of prompt caching are where this feature really shines for applications with repetitive content. Here's the detailed cost structure:

1. Cache Write (First-time Processing): When content is processed and written to the cache for the first time, you pay a small premium over regular processing. For third-party models like Anthropic's Claude on Bedrock, the cache write cost is approximately 25% higher than the standard input token price. For Amazon's own models, there's currently (as of March 2025) no extra charge for cache writes.

2. Cache Read (Subsequent Reuse): The big savings come from cache reads. When you reuse cached content, you pay only about 10% of the normal input token price, a 90% discount compared to processing those tokens from scratch.

3. Storage Costs: There are no separate storage fees for keeping data in the cache. You only pay the read/write token fees described above.

This pricing structure reflects the real computational cost difference between processing tokens from scratch and reusing pre-computed representations, which I talked in detail about at the beginning of the article.

Let's see some example numbers.

Imagine you're building a financial document analysis system that allows users to upload quarterly reports (average 30,000 tokens) and ask multiple questions about them. Each user session involves around 8 questions about the same document. Without caching, each question would require reprocessing the entire document.

Let's calculate the costs using the newly-released Claude 3.7 Sonnet pricing (Claude 3.5 Sonnet has the same price):

• Input tokens (no cache): $0.003 / 1k tokens

• Output tokens: $0.015 / 1k tokens (same price with or without cache)

• Input tokens (cache write): $0.00375 / 1k tokens (25% more expensive)

• Input tokens (cache read): $0.0003 / 1k tokens (90% cheaper)

Without Caching:

• Document processing per question: 30,000 tokens × $0.003/1000 = $0.09 per question

• Total for 8 questions: 8 × $0.09 = $0.72

With Caching:

• First question (cache write): 30,000 tokens × $0.00375/1000 = $0.1125

• Subsequent 7 questions: 7 × 30,000 tokens × $0.0003/1000 = $0.063

• Total: $0.1125 + $0.063 = $0.1755

That's a 75.6% cost reduction for a single user session. Now scale that to an enterprise with thousands of users and documents, and the savings add up to a lot. For a system processing 10,000 documents per month with this pattern, you'd save over $5,400 monthly.

But it's not always that simple. Some scenarios where caching might not help much:

• Single-use content: If each document is only analyzed once, the cache write premium actually increases your cost by 25% 🫠.

• Tiny prompts: For very small prompts below the minimum token threshold caching doesn't activate, so you don't save anything (nor pay extra).

• Long gaps between requests: If users typically wait more than 5 minutes between questions, the cache expires and you lose the benefit. Of course you still pay for the cache write, so again your costs increase by 25%.

• High cache miss rate: If your application frequently generates slightly different versions of prompts that can't share cache entries, you'll pay for cache writes without getting read benefits.

To effectively measure your actual savings, implement logging for the cacheReadInputTokens and cacheWriteInputTokens metrics from your API responses. Over time, this data can help you optimize your caching strategy and quantify the ROI.

There's another economic benefit worth mentioning: by reducing response latency, caching can also improve user experience. This secondary economic benefit doesn't show up directly in the token costs but can significantly impact your app, especially since end users’ most common complaint about AI applications (in my experience) is that they take too long to respond.

Performance Optimization

As I mentioned, prompt caching can have a significant impact on performance. According to AWS, cached content can be processed up to 85% faster than uncached content. This translates directly to lower latency for your users.

The performance gain scales with the size of the cached content, following an interesting pattern. Caching a few hundred tokens might save tens of milliseconds, while caching thousands of tokens can reduce response times by seconds or even tens of seconds for very large prompts. This non-linear relationship occurs because the token processing time in LLMs isn't perfectly linear, there are fixed overheads and optimizations that vary based on the total workload.

Here's a technical insight you might not find in the documentation: the performance benefit isn't just from skipping token processing, it also comes from avoiding the initial model loading and warmup. When a model starts processing a prompt, there's a "ramp-up" period where tensor operations aren't fully optimized. By jumping straight to a cached state, you skip this ramp-up, giving an additional performance boost beyond the raw token processing time.

Tips to optimize prompt structure for maximum performance

1. Place cache checkpoints at logical boundaries in your prompt, such as after system instructions or reference documents but before user queries

2. Ensure that static content comes before dynamic content in your prompts

3. Meet the minimum token threshold for your model before inserting a checkpoint

4. Structure multi-turn conversations to leverage previously cached content

A pitfall I've encountered is what I call "cache fragmentation", creating slightly different versions of similar prompts that can't benefit from the same cache entry. For example, if you include timestamps or request IDs in your prompt prefix, you'll create a unique cache entry for each request, effectively nullifying the benefits of caching. To avoid this, standardize your prompt templates and ensure that fixed content is consistent across requests.

For example, here's a simple prompt that contains variable data where you'd tend to place it:

You will be presented with 3 articles about AWS. You must answer the user's questions about them. The user is a cloud engineer, so make sure you adopt an appropriate tone in your responses.

See the variables? 3 articles about AWS, and the user is a cloud engineer. So if you later need to send 4 articles instead of 3, or they're about AI instead of AWS, or the user is a sales rep instead of an engineer, you can't reuse this from the cache. Here's how you can rewrite it instead:

You will be presented with some articles about a specific topic, which I'll specify at the end of this prompt. You must answer the user's questions about them. The user has a certain role, which I'll specify at the end of this prompt, so make sure you adopt an appropriate tone in your responses.

[checkpoint]

topic: AWS
user role: cloud engineer

Now you can put a checkpoint at [checkpoint] and reuse everything that comes before it as a cached prompt. Of course this example is way too brief to cache, but I hope you get the idea.

Implementation Patterns for Common Scenarios

Different use cases require different implementation approaches for prompt caching. Let's look at some common patterns I've found effective:

Multi-Turn Conversations

For conversational applications, cache the system prompt and conversation history to avoid reprocessing previous exchanges. In a multi-turn implementation:

def handle_conversation(conversation_id, user_message):
    # Retrieve conversation history
    history = get_conversation_history(conversation_id)
    
    # Structure messages with cache points
    messages = [
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "system", "content": {"cachePoint": {"type": "default"}}}
    ]
    
    # Add previous exchanges with a cache point after history
    for msg in history:
        messages.append(msg)
    
    if history:
        messages.append({"role": "user", "content": {"cachePoint": {"type": "default"}}})
    
    # Add the new user message
    messages.append({"role": "user", "content": user_message})
    
    # Call Bedrock with caching enabled
    response = bedrock.converse(
        modelId=MODEL_ID,
        messages=messages
    )
    
    # Save the exchange to history
    update_conversation_history(conversation_id, 
                               {"role": "user", "content": user_message},
                               {"role": "assistant", "content": response["output"]})
    
    return response["output"]

This approach places checkpoints after the system prompt and conversation history, allowing the model to skip redundant processing of earlier messages. One thing to watch out for is the context window limit, if the history grows too large, you'll need to summarize or truncate it while maintaining the cache checkpoints. And if you summarize it, make sure you drop the cache by starting a new conversation! I've made that mistake of adding the summary but also keeping the old stuff 🤦.

Document Q&A

For document-centric applications (i.e. the first thing that comes to mind when I say RAG), cache the document content to enable fast querying:

def document_qa(document_id, query):
    # Retrieve document content
    document = get_document_content(document_id)
    
    # Structure request with document caching
    messages = [
        {"role": "user", "content": [
            {"text": "I want to ask questions about this document:\n\n" + document},
            {"cachePoint": {"type": "default"}},
            {"text": query}
        ]}
    ]
    
    # Call Bedrock with caching enabled
    response = bedrock.converse(
        modelId=MODEL_ID,
        messages=messages
    )
    
    return response["output"]

A non-obvious optimization here: for very large documents, you might need to split them into chunks with multiple cache points. Claude 3.5 supports up to 4 checkpoints, so you could structure a long document with checkpoints after each quarter, allowing partial reuse even if some sections change. If nothing changes, don't bother, one huge checkpoint and 4 smaller checkpoints have the same performance.

Coding Assistant

For coding assistants that analyze codebases:

def code_assistant(repository_id, file_paths, query):
    # Retrieve code files
    code_context = ""
    for path in file_paths:
        code = get_file_content(repository_id, path)
        code_context += f"File: {path}\n```\n{code}\n```\n\n"
    
    # Structure request with code caching
    messages = [
        {"role": "system", "content": "You are a coding assistant that helps with programming tasks."},
        {"role": "user", "content": [
            {"text": "Here is the code to analyze:\n\n" + code_context},
            {"cachePoint": {"type": "default"}},
            {"text": query}
        ]}
    ]
    
    # Call Bedrock with caching enabled
    response = bedrock.converse(
        modelId=MODEL_ID,
        messages=messages
    )
    
    return response["output"]

An important note for code analysis: because code often changes incrementally, consider version-specific caching strategies. For example, you might include a git commit hash in your cache key to ensure you're not using outdated cached representations for modified code.

To be honest I haven't found a good way to use cache for this. My conclusion is that I'd need to be able to predict which files are less likely to change so I can place them in the cache. This sounds partially doable with some pre-processing of the files and the user's query, but I haven't tested it, nor explored other ideas. Too busy writing!

Debugging Cache Issues

When things go wrong with prompt caching, diagnosing the problem can be tricky without knowing what to look for. Here are the most common issues I've encountered and how to resolve them:

Identifying Cache Misses

The first step in debugging is determining whether your cache is being used at all. The most reliable method is to check the response metrics:

def is_cache_hit(response):
    usage = response.get("usage", {})
    read_tokens = usage.get("cacheReadInputTokens", 0)
    return read_tokens > 0

If this function returns False, your cache isn't being hit. The most common reasons are:

1. Cache expiration: The 5-minute TTL elapsed between requests

2. Prompt mismatch: The prefix text doesn't exactly match the cached version

3. Token threshold not met: You're trying to cache a segment smaller than the minimum requirement

4. Cache point not properly placed: The cachePoint marker is missing or incorrectly formatted

Conclusion: Please Use Prompt Caching Often (not always)

Prompt caching should be a standard consideration in your design process. It's particularly valuable for applications with these characteristics:

• Multi-turn conversations with consistent system prompts

• Document-centric analysis where users ask multiple questions about the same content

• Coding assistants that need to reference the same codebase repeatedly (though this needs a bit more work)

• Any workflow where large static context is combined with smaller dynamic queries

When not to use it: Workloads without a sufficiently long, reusable prompt prefix or enough cache hits to justify the write cost. Supported InvokeModel requests can benefit too.

Tips when using caching:

• Design your prompts with caching in mind from the beginning. For example, move variables to the end.

• Monitor cache performance and cost metrics to validate your approach. Remember that writing the cache costs 25% more, so if you're not benefitting you're overpaying.

• Plan for cache misses and expirations. They will happen more than you think.

That's it. Sorry I took a bit long to say that, but I hope you found the inner workings interesting!