Monitoring Amazon Bedrock: find out before your users do

The problem

You’ve shipped a feature backed by a large language model. It works in the demo, it works in your test account, and then it goes to production and somebody asks the question every engineering lead eventually asks: “how do we know if it’s actually working right now?”

With a traditional API you’d reach for the usual suspects (request count, error rate, p99 latency) and call it a day. Features backed by an LLM need all of that, plus a second layer that doesn’t exist in a typical REST service:

• Latency isn’t one number. A streaming response has a “time to first token” (how long before the user sees anything) and a “tokens per second” throughput once it starts. A feature can look fine on p99 total latency while feeling broken because the first token takes three seconds to show up.
• Cost is per request and variable. Every call has an input token count and an output token count, and they don’t move together. A user pasting in a long question costs more on the input side; a verbose answer costs more on the output side. There’s no fixed “cost per API call” the way there is for most backend services.
• Failure modes are different. Throttling, validation errors caused by context length, and model timeouts all look different from a 500 error, and they need different responses. A throttle means back off and retry. A validation error means your prompt construction has a bug. A timeout on a long generation might just mean the model is doing what you asked, slowly.
• The same infrastructure serves very different workloads. One feature might be a quick classification call, another a lengthy explanation spanning several paragraphs. Averaging their latency and cost together produces a number that describes neither.

None of this is optional to monitor once real users are involved. Slow responses, silent cost overruns, and unnoticed error spikes all show up in user experience. The only question is whether you find out yourself or you hear about them from your users and risk losing their trust. Observability, in this context, is not an optional engineering practice, it is the difference between shipping an LLM feature you can operate and shipping one you are just hoping keeps working.

A few scenarios where this actually bites you

The silent cost creep. A prompt template gets updated to include more context “just to be safe.” Nobody notices, because the feature still works; it is just quietly 40% more expensive per request. Without per request cost tracked and broken out by feature and prompt version, this shows up a month later as a line item in the AWS bill, with no way to tell which change caused it.

The regional latency surprise. The same feature, called from two different regions, does not perform the same. If you’re not tagging your metrics by region, “latency is fine” and “latency is fine for one region and bad for another” look identical on a single aggregated chart.

The staged rollout that isn’t. You ship v2.5.0-beta to staging before promoting it. If staging and production traffic aren’t distinguishable in your metrics, you can’t actually compare the beta against the current production behavior side by side; you’re stuck flipping between two different time windows and hoping nothing else changed in between.

The retry storm. A model starts throttling under load. Your code retries, which is correct, but if retries aren’t tracked as their own metric, a spike in retries just looks like a spike in latency. By the time anyone traces it back to throttling, the root cause investigation has already burned an hour.

The error that isn’t loud enough to page anyone. A 1% error rate on a feature with little traffic might be five errors a day: not enough to trip a naive “errors > 0” alarm, but plenty to matter if it’s always the same feature failing for always the same reason. You only see that pattern if errors are broken out by feature and error type, not lumped into one counter.

Every one of these is fixable after the fact by digging through logs. The point of observability is that you don’t have to wait to be asked.

Why CloudWatch is a solid fit for Bedrock

Bedrock already writes some of this for you, for free. Every model invocation emits metrics into the AWS/Bedrock namespace, including invocation counts, latency, throttle counts, and cache read/write tokens, with no code changes required. That’s a real head start most setups that host and serve their own models don’t get out of the box.

The second half of the picture (the things that are specific to your application rather than the model call itself, such as which feature, which prompt version, cost per request, and time to first token) is exactly what CloudWatch custom metrics are built for. You publish a handful of PutMetricData calls from wherever you’re calling Bedrock, tag them with dimensions that matter to your product, and they show up right next to the native metrics in the same namespace and dimension model.

A few things make CloudWatch specifically a good match here rather than just an available one:

• It’s already the place your Lambda/ECS/EKS logs live. There’s no second system to stand up, no second vendor to onboard, and no separate place to go looking during an incident.
• Model Invocation Logging is a feature that’s already built in, not something you build. Turn it on once at the account and region level, and every Bedrock call, across every service that calls it, gets logged to CloudWatch Logs and S3, full request and response payloads included, without touching application code.
• Dimensions have a free form. You’re not locked into a fixed schema. Whatever you want to slice by, such as environment, feature, prompt version, or customer tier, just becomes a dimension key on the metric you publish.
• CloudWatch Logs Insights turns unstructured logs into ad hoc queries that feel similar to SQL. “Show me every error for this feature in the last hour, grouped by error type” is a query only five lines long, run against the same logs you’re already writing to stdout.
• It composes with everything else in the account. Alarms, dashboards, anomaly detection, and observability across accounts all already understand CloudWatch metrics, so you’re not building a monitoring island.

The tradeoff is that CloudWatch custom metrics aren’t free once you have a lot of dimension combinations (more on that below), but for the volume most teams are dealing with early on, it’s a rounding error compared to the model invocation cost itself.

The solution

The reference implementation is on GitHub at saeed2402/bedrock-observability — a small, runnable setup: a Lambda function that calls Bedrock the way a real backend service would, and a CloudWatch dashboard that visualizes the result. It’s built to be read end to end in about fifteen minutes, then adapted.

Architecture:

Lambda (traffic generator, standing in for a real service)
    |
    +=> Bedrock converse_stream (openai.gpt-oss-120b-1:0)
    |         |
    |         +=> native AWS/Bedrock metrics (throttles, cache tokens, ...)
    |         +=> invocation logs => CloudWatch Logs + S3
    |
    +=> CloudWatch custom metrics (BedrockDemo namespace)
              |
              +=> CloudWatch dashboard

What gets tracked, and where it comes from:

Every custom metric carries seven dimensions: Environment, Region, Model, ServiceName, ServiceVersion, Feature, PromptVersion. That’s the part that actually answers the scenarios above. Questions like “is latency worse in ap-southeast-2,” “is the beta version more expensive than prod,” and “which feature is throwing all the errors” are all just a metric selection away, not a new instrumentation effort.

A couple of implementation details worth calling out because they came from actually running this against a live Bedrock model rather than just reading the docs:

• Cost and cache savings are published as USD times 1000 (“millicents”). CloudWatch’s vertical axis doesn’t render fractions of a cent legibly, so the raw values get scaled up for anything under a cent and get scaled back down when you read them.
• The IAM role Bedrock assumes for invocation logging needs source account and source ARN conditions in its trust policy. Without them, the role can technically be assumed by Bedrock in any account, which is exactly the kind of overly broad trust relationship security tooling flags, and rightly so.
• A small deliberate path for injecting errors exists in the demo Lambda (about 1% of calls send an invalid maxTokens) purely so the error rate widget isn’t a permanent flat line when you’re first standing this up. It’s an obvious thing to strip out once you’re wiring real traffic through this pattern.

Using it in production

The repo ships as three pieces: lambda/lambda_function.py (the Bedrock caller and metric publisher), infra/dashboard.json (the dashboard definition), and infra/deploy.sh plus infra/cleanup.sh (idempotent setup and teardown). Running ./infra/deploy.sh creates the S3 bucket and IAM role for Bedrock invocation logging, turns logging on, creates the Lambda’s execution role, deploys the function, and publishes the dashboard, in that order. It can safely be run again if any step needs to be repeated.

To go from this reference implementation to something running in front of real traffic:

1. Replace the traffic generator with your actual call site. The part that matters is call_bedrock() and publish_metrics() in lambda_function.py. Lift that pattern into whatever service actually calls Bedrock (Lambda, ECS task, EKS pod, it doesn’t matter), and drop the scenario generation scaffolding.
2. Pick your dimensions before you pick your metric names. The seven dimensions used here aren’t sacred; they’re simply the ones that mattered for this scenario. Decide what you’ll actually want to slice by in an incident (customer tier, model version, deployment region) and bake those in from day one, because adding a new dimension later means every existing time series is a different metric than the new one.
3. Turn on Model Invocation Logging at the account level once, rather than once per service. It’s a single setting that applies to the whole account and region, so if multiple services call Bedrock, they all benefit from the same log group and S3 archive without any of them needing to configure it themselves.
4. Set alarms on the metrics that predict pain for users, not just the ones that are easy to alarm on. Time to first token measured against an SLA threshold, and error rate broken out by feature, are both more actionable than a raw invocation count. The dashboard in this repo includes an SLA annotation line on the TTFT widget as a starting point; wire a real cloudwatch put-metric-alarm against it once you trust the numbers.
5. Watch your dimension cardinality. Custom metrics are billed per metric and dimension combination, not per data point. Seven dimensions with a handful of values each is fine, but a customer ID dimension with ten thousand distinct values is a different bill entirely. If you need granularity down to the individual customer, that’s usually a job for logs and Logs Insights queries rather than a dimension on every metric.
6. Query the logs, don’t just stare at the dashboard. The dashboard answers “is something wrong.” CloudWatch Logs Insights against the invocation log group answers “what exactly happened at 3:14pm to this one request,” including token counts, model ID, and, if you’ve enabled it, the actual prompt and response. Keep both habits.
7. Check the cost of the observability itself. At the traffic levels typical of a demo, this setup runs somewhere around $25 a month for the dashboard, custom metrics, and log storage combined, which is worth knowing before you scale dimension cardinality or invocation volume by 100x and forget that the monitoring bill scales too.

None of this is exotic. It’s the same discipline you’d apply to any production service, namely knowing your latency, your errors, and your cost, applied to the two or three things about LLM calls that don’t fit the old templates. The value isn’t in any single metric; it’s in having all of them, tagged consistently, sitting in the same place you already look when something’s wrong.

The code is at github.com/saeed2402/bedrock-observability — clone it, run ./infra/deploy.sh, and you’ll have a live dashboard in a few minutes.