How Routing Trees Solve the Enterprise AI Context Problem

What you need to know: A routing tree replaces a single oversized system prompt with a graph of small, single-purpose instruction nodes, so an enterprise agent loads only the instructions relevant to the current decision. We adopted one after our own AI Agent crossed 400K tokens of context per turn and still hallucinated, skipped steps, and lost the thread. Moving to a routing tree dropped per-message token usage by roughly 95% and took reliability from "uncomfortable demo" to "production-ready."

Why One Giant Prompt Stops Scaling

Levelpath builds AI Agents that run real procurement work for enterprise teams: intake, sourcing, supplier questions, policy lookups, and approvals. Each customer brings a tangle of internal rules, and the natural first move is to write them all into one instruction file. When we onboarded an enterprise customer earlier this year, that file crossed 400,000 tokens, and every new policy PDF or contact list only pushed it higher. The giant prompt approach breaks down in three ways:

• Cost and latency. A 400K-token prompt is consistently slow on every turn, no matter how trivial the question. "Who do I contact about category X?" pays the same per-token tax as a full sourcing workflow, and the prompt only grows.
• Relevance dilution. Almost none of the prompt is relevant to any single message. Long-context models can hold that much in the context window, but fitting a policy library and using the right rule at the right moment are different things, and the model often latches onto a rule that only resembles the situation.
• Behavioral drift. Asked to hold an entire rule library in working memory, the model fills gaps itself: contact names that appear in no source file, supplier names stitched from fragments, workflow steps in the wrong order. Each slip is plausible, and for a regulated enterprise, completely unacceptable.

Our customer hit all three at once. The Agent did not meet their standard, and because "unreliable" is the kiss of death for an enterprise AI deployment, the relationship could be at risk.

The Core Idea: Route, Do Not Stuff

Instead of handing the model everything on every turn, Levelpath now gives it the slice it needs right now, plus a small piece of logic that decides which slice to load next. We do this with a routing tree: a directed graph of instruction nodes, where each node holds a focused chunk of instructions for one decision or one terminal action. The agent starts at the root, traverses node by node, and stops at a terminal, the equivalent of saying "you are going to system X with this context, here is the link, and here is why." At any turn the model sees only the current node plus a few shared sections it always needs, so the 400K-token monolith becomes a sequence of focused 5-to-15K-token prompts.

How the Agent Moves Through the Tree

The model starts in the root node, which holds one instruction such as "classify the request into one of N intent categories." From there, one of two things happens:

1. The message is specific enough. The model calls an "advance" tool to move to the child node for that category, and the next turn loads that child's instructions. Nothing is asked of the user; nothing is made up.
2. The message is ambiguous. The model asks a single focused follow-up question, then advances once the user answers.

This repeats at every node. The model never reasons over the whole tree; it just decides locally whether the context answers the current node's question. A user who types a fully-specified request (region, category, spend, and supplier) can travel root to terminal in one round trip, while less-specified messages cost an extra turn only for the pieces the route actually needs.

Designing the Tree Before You Build It

The mechanical work of defining nodes and wiring edges is the easy half. The hard half is knowing what the tree should look like, and the single most expensive mistake we made was building against a diagram the customer was still revising. Every change rippled outward, invalidating individual nodes, whole branches, and sometimes the top-level categorization, and the implementation time grew exponentially as the diagram moved underneath us. Finalize the process flow diagram before you write a single node, or you will spend more time un-building than building. Once it is signed off, work in this order:

1. Enumerate the terminal outcomes. Every distinct "route to system X with context Y" is a terminal node, and listing them up front seeds the test plan.
2. Group terminals into intents. Most agents have ten or twenty high-level intents even with a hundred-plus terminals.
3. Work backwards from each terminal to derive the chain of decisions that leads to it.
4. Define each node by the single decision it owns, with a crisp question like "what region is this request for?"
5. Merge similar nodes into converging nodes, so a step that recurs across routes lives in exactly one place.

This is more product work than engineering work. It’s extremely useful to have someone on the building team who has genuinely absorbed the customer's business, enough to catch a shaky categorization early.

The Optimizations That Kept It Maintainable

A few structural patterns kept the tree maintainable as it grew:

• Shared vs. attachable nodes: Always-relevant sections (context schema, guardrails, and formatting) auto-load, while reusable-but-occasional sections attach only where a node references them.
• File attachments per node: Each node carries only the PDFs, CSVs, and lookup tables it needs, OCR'd to text at upload. Attach too many large PDFs to one node and you recreate the original problem, so we split overloaded nodes into a small router plus one terminal per document.
• Data-driven decisions via CSVs: Routing that is really table lookup (category to workflow, region to reviewer) lives in attached CSVs, so a business user can re-upload the file when the data changes and the routing follows.
• Rerouting: When a user changes their mind mid-conversation, we reset to an earlier node and selectively clear the context fields that no longer apply while keeping the ones that still do.
• Structured state, not chat history: Each node writes the facts it establishes into a shared context object, so downstream nodes read a value instead of re-inferring it and quietly changing their mind.
• Traceable changes and updates: The whole tree serializes to YAML, which makes every change a reviewable diff.

Testing a System That Talks to an LLM

Once the tree passed a couple dozen nodes, manual testing became impossible, so we built a runner that exercises every path with scripted inputs and checks the terminal, the route, and the collected context. A few lessons stood out:

• Test cases come from the diagram, not the tree. A failing test might mean the tree is wrong, or it might mean the requirements were, which is why the signed-off diagram is load-bearing.
• Check both route and content. Landing on the right terminal does not guarantee the response is correct, so we verify the substance too.
• Track flakiness explicitly. With an LLM in the loop, a test can pass 99 times and fail the hundredth. The fix is to study the failure, not silence the test.

The Challenges That Remain

The routing tree solved our scaling problem, but it did not make everything easy. These are the parts that still take real work:

• Building before requirements are final is the costliest mistake. "We will figure that out as we go" is the most expensive sentence in this workflow.
• Initial categorization is load-bearing. Get the top-level intents wrong and almost every later restructure traces back to it.
• The LLM is still an LLM. It will occasionally pick the wrong branch or hallucinate a name from a table it just read, so the system has to be graceful about that residual. Rerouting is the sharpest version, since making the model forget exactly the right context is genuinely hard.
• Auto-advance is partly out of your hands. Whether the model advances or asks is its own tool-calling decision. You can influence it, not make it deterministic.
• Knowing when to say "I do not know." Calibrating the model to bow out gracefully, without bailing too early or forcing a bad match, is one of the more delicate knobs.
• Model swaps are not free. Upgrading the underlying model can quietly change how the same tree behaves, so we re-run the full suite every time and have caught real regressions.

The Results: 95% Fewer Tokens and a Reliable Agent

Per-message token usage dropped by roughly 95%, and the unreliability that was sinking the agent went away. It now handles the full range of request types the monolith was meant to cover, with a structure we can extend (add a category, a terminal, or a policy document) without ripping anything up. The final shape is 72 nodes and 84 edges, with a small auto-load core, around 27 terminal outcomes, and roughly a quarter of all nodes converging across multiple routes. For a procurement team, that is the difference between an agent they trust to route a real sourcing request and one they quietly stop using. The bigger payoff goes beyond this one account. "Routing tree" is now a pattern we reach for whenever this shape of problem recurs, whether that is the next customer arriving with a megaprompt or an internal task with the same structure.

See what this looks like in production. Book a demo to watch Levelpath's AI Agents run enterprise procurement end to end.

*The work described in this blog was the joint effort of Nikola Upite and Anna Bicevska.