How a lightweight, open prompt router cuts LLM costs by sending easy questions to cheap models, escalating only when needed, and training itself to route better over time.

Solution github

The problem: every prompt pays the premium price

If you run an application on large language models, you have probably noticed something uncomfortable in your bill. The prompt “What’s 2 + 2?” costs you the same per token as “Design a fault tolerant multi region payments architecture.” Both go to the most capable model you have, because that is the safe choice. Nobody wants a customer to receive a weak answer to a hard question.

The result is that the majority of requests, which in most applications are short, factual, and simple, are answered by a model that is orders of magnitude more expensive than it needs to be. Flip the setup around and send everything to the cheap model, and you get the opposite failure: complex questions receive shallow answers, users lose trust, and someone on your team ends up manually triaging quality complaints.

Amazon Bedrock offers a native feature for this called Intelligent Prompt Routing, and it works well within its boundaries. The catch is the boundaries. It only pairs two models from the same family, such as two Claude models or two Llama models. If you want to route between an Amazon Nova model and a Claude model, or between an OpenAI open weight model and DeepSeek, the native router cannot help you. And because its quality prediction model is managed by AWS, you cannot inspect or adjust why a given prompt was routed where it was.

The solution: a transparent router with a learning loop

The bedrock-cross-provider-prompt-router is a small, dependency light Python library plus an optional AWS pipeline that fills this gap. It routes each request between any two Bedrock models you choose, from any providers, using rules you can read, log, and change. Here is the whole idea in one paragraph.

Before any model is called, the router runs a few local checks on the prompt: structural signals such as multiple questions and step words, length combined with structure, the token volume of the conversation history, and a curated list of complexity keywords matched on word boundaries. These checks cost nothing because they are plain string and regex operations. Simple prompts go to the cheap model. Complex prompts go straight to the strong model. If the cheap model answers but the answer looks unreliable, meaning it came back empty or contains hedging language like “I don’t have access to” or “I’m unable to,” the router automatically retries the same prompt on the strong model and returns that answer instead. In the common case a request costs one model call. A second call happens only when it is genuinely needed. Here’s a logival view of how this solution routes and escalates prompt:

That alone is useful, but the more interesting part is the learning loop. Every routing decision can be logged to CloudWatch as a structured metric line, recording the prompt’s features and whether the cheap tier ended up needing an escalation. A scheduled job exports those outcomes as training data, and a small scikit-learn classifier is trained to predict, up front, whether a prompt will need the strong model. The new classifier is published only if it beats the currently deployed one on held out data, so a noisy training run can never make routing worse. The router picks up the new model automatically and falls back to the hand written rules whenever no trained model is available. Over time, the router gets better at recognizing hard prompts before wasting a cheap tier attempt on them, which removes both the extra cost and the extra latency of failed first tries. Below is the solution architecture diagram:

Everything deploys with a single script that creates the IAM roles, two Lambda functions, an S3 bucket, and a weekly schedule, then smoke tests the whole path. A matching cleanup script removes every trace. There are no account specific values anywhere in the repository, and the routing logic itself is fully inspectable: every response carries a record of exactly which signals drove the decision.

Where this pays off: five industry scenarios

Customer support platforms. Support traffic is the textbook case of mixed complexity. Password resets, order status checks, and store hours are trivially answerable by a small model, while multi step billing disputes or technical troubleshooting need a strong one. A router that sends the easy majority to a cheap tier can cut inference spend dramatically, and the escalation check acts as a safety net when the cheap model hedges instead of answering.

Financial services. Banks and fintechs handle a stream of simple balance and definition questions alongside genuinely hard ones about regulation, contracts, and risk. The keyword signals in this router already treat terms like legal, compliance, and regulation as strong tier triggers, and because the rules are transparent, a compliance team can review and extend exactly what routes where. That auditability is something a managed black box router cannot offer.

Healthcare administration. Appointment scheduling, coverage lookups, and form guidance are high volume and low complexity, while clinical documentation summaries and prior authorization narratives demand a capable model. Routing the administrative bulk to a cheap tier keeps costs predictable, and the logged routing signals provide a clear paper trail for why each request was handled by which model.

E-commerce and retail. Product question answering, review summarization, and catalog enrichment run at enormous volume. Most of it is simple, some of it is not, and margins are thin enough that a four times cost difference per token matters. The learning loop shines here: with high traffic, the classifier accumulates training data quickly and adapts to what customers actually ask, rather than what the initial rules guessed they would ask.

Software and developer tools. Coding assistants and internal engineering bots see everything from “what does this flag do” to “help me debug this race condition.” The router’s structural score catches multi part requests, its debug and architecture keywords catch the hard cases, and teams can pair an inexpensive general model with a premium coding model from a different provider, which is precisely the pairing the native Bedrock router does not support.

When to use it

Use this router when you are pairing models from different families or providers, when you want routing logic you can read, test, and tune rather than a managed prediction model, or when you are experimenting with newer models that native routing does not yet cover. It is also a good fit when you want routing quality to improve from your own traffic, since the retraining pipeline turns your real outcomes into a classifier tailored to your workload. Because the core router is a single Python class with no deployed infrastructure required, the cost of trying it is one import and two model IDs.

When not to use it

If your two models are in the same family, are supported by Bedrock Intelligent Prompt Routing and you simply want managed quality prediction with zero maintenance, Bedrock’s native Intelligent Prompt Routing is the simpler choice. If nearly all of your traffic is uniformly complex, routing adds a little machinery for little savings, and you should just call the strong model. The confidence check works by detecting empty responses and hedging phrasing, so if you route to a cheap model whose refusal style differs greatly from what the patterns cover, plan to run a short calibration test and extend the pattern list. The learning loop needs traffic to earn its keep: below roughly fifty logged requests the trainer deliberately refuses to run, and its escalation label is a proxy that inherits the biases of whatever the heuristic routed during data collection. Finally, this router uses the non streaming Converse API. A streaming chat interface would need adaptation, because you cannot check a response for hedging language until the stream completes.

Try it

The repository ships with runnable examples, a fully mocked test suite, an implementation guide covering the routing internals and real world model ID gotchas, and a retraining guide that walks the full learning loop from first log line to a published classifier. Deployment into your own AWS account is one script with two required arguments, and teardown is one more. Point it at a cheap model and a strong model your account can invoke, send it your real traffic, and let your bill tell you whether it earned its place.