What you need to know: When an agentic loop breaks a task into narrow steps, each with one job and a clear definition of done, it removes most of what separates one model from another. Models then land close together on outcome, and model selection stops being a capability question and becomes a question about speed, cost, and rule-following. Across five model configurations on real tickets, the loop decided whether the work got finished, not the model.
Levelpath’s Agentic Loop
Levelpath builds much of its own product with an agentic software factory that we call the Agentic Loop. A ticket goes in and a finished, review-ready code change comes out. A human merges it, because merging is intentionally not part of the loop's job.
Between sits a chain of specialized agents, each with one job. ‘Quality check’ triages the ticket and picks the route. ‘Build or fix’ writes the code. ‘QA execution’ assesses whether it worked. ‘Signoff assurance’ independently audits that assessment. A coordinating agent never writes code. It routes, records, and decides the next step. Every step is a model call, and which model runs each one is a recurring decision with money attached.
Why Public Benchmarks Do Not Answer the Question
The obvious approach is to read the model leaderboards and pick whoever is on top. We tried. Benchmarks score a model working alone on a small, self-contained, public task. Three gaps made the scores unusable:
- They measure the wrong thing: We grade a code change that has to pass tests written by someone else, satisfy a style checker, respect our codebase conventions, and stay inside the ticket's scope. No benchmark scores that combination.
- They ignore what actually differs: Cost and elapsed time barely appear on leaderboards, and inside a loop making many calls per ticket, both compound.
- They move for unrelated reasons: Scores shift with every model release, and whether our loop still produces mergeable code is answerable only by running it.
The Core Idea: Narrow the Problem, Then Shop on Cost
Models vary considerably on generic coding benchmarks, and inside our loop they land close together in both directions: on the tickets our loop handles well, every model resolved them completely, and on the tickets it handles badly, no model did.
A well-built agentic loop removes choices. An open-ended ticket can be done well or badly in many ways, and that room is where a stronger model pulls ahead. Our loop closes it: each step arrives with the context gathered, the scope settled, and the conventions checked, so the question in front of any model is small and precisely stated.
We made the problem artificially easier, and that is the point. Once most models clear the bar, capability stops being the differentiator, and what remains are the axes a business feels: how fast, how cheap, and how reliably the model follows the rules. Most of our steps run on a cheap model for that reason, and run metrics record which model each agent used, so cost and quality get correlated.
Narrowing the problem pays off twice: the work gets more reliable, and the model becomes a commodity input you can shop for. We took the same lesson from routing trees, where replacing one oversized prompt with small single-purpose ones cut per-message tokens by roughly 95%. Structure buys more than scale does.
Where Cheap Is the Wrong Answer
One step does not follow the cheap default. Signoff assurance audits the QA verdict and records a second opinion without changing what happens next, on a strong model from a different family than the verdict it checks.
The constraint is not the price tier. It is separation. A reviewer sharing its subject's model family inherits the blind spots it exists to catch, so if the QA steps ever move onto that family, signoff moves off it. Its per-run cost is cents, so cost is never a reason to downgrade.
The rule we would offer anyone building one: shop on cost everywhere except the steps whose whole job is to catch what another model missed. Those are bought on independence alone.
Grading Cheapest Signal First
The eval framework runs three suites, ordered so the expensive ones run only when needed.
- A contract floor: Text searches over the playbooks, checking that every hard gate, boundary rule, and report field is still named. Free, safe on every commit, and it catches the most common regression: a playbook edit that drops a gate.
- Graded outputs: Graders score each dry run's changed files, report, and metrics against known-good answers across seven dimensions, including scope discipline and security. Only rows needing judgment reach a grader agent.
- A whole-loop run: A real ticket goes through the whole pipeline, graded against the tests from the human-written fix: tests that fail before the fix and pass after it, applied at grading time, never shown to the agent.
Fixtures That Can Tell Good From Bad
A suite where everything passes has stopped producing information.
Test cases with a trap built in. A test that every agent passes tells you nothing about which agent is better. The useful ones have a tempting wrong answer built into them. One of ours rewards an agent for leaving alone an unrelated bug it spots along the way, rather than fixing it and expanding the change. Another looks like a routine data-access request, where the obvious implementation would let one customer see another customer's data.
Some fixtures have "I cannot do this" as the correct answer. An agent that invents a schema or fakes a fix fails, because honest reporting is a graded behavior when a human is the merge gate.
A deliberately sabotaged playbook must fail the suite. Without that check, a green result could just mean the suite cannot tell good from bad.
What a Recent Batch Showed
We ran a recent batch at Levelpath with five model configurations, two real tickets, and one trial each. Cost is estimated from reported token usage rather than billed, so the tables rank configurations rather than price them. On the ticket the loop handles well, every configuration resolved it, with every test passing.
The fastest configuration was also among the cheapest, finishing in under half the time of the slowest. One of the five flagged itself for review anyway after passing every test.
On the harder ticket, nothing was resolved. Every configuration landed partial, inside a two-point band.
The most expensive configuration bought one percentage point over the cheapest, and the second most expensive scored a point below it. With two tickets, anything smaller than a full ticket's worth of outcome, roughly fifty points, cannot be separated from noise. That point is a rounding error at the top of the price range.
Within this batch, model choice did not determine whether the ticket got finished. The loop did. When the loop fit the problem, everything was resolved. When it did not, nothing was, and no amount of model spend closed the gap.
What This Means for Engineering Teams
The ordering is the transferable part. Narrow first, evaluate second. Break the work down until every model call has one job and a checkable definition of done, and only then stand up the eval. Compare models against a loose pipeline and you mostly measure variance.
Most teams treat the model as the lever and the pipeline as scaffolding. Invert it. If you build one piece first, make it the grader. Once you can reliably tell whether an answer is right, the loop can retry and iterate toward the right answer, making the quality of any single model attempt much less important.
The audit trail comes free. A loop with named steps and explicit gates already records what ran, on which model, and why it advanced, which is what anyone reviewing an agentic system in an enterprise asks to see.
See what this looks like in production. Book a demo to watch Levelpath's AI Agents run enterprise procurement end to end.
