Your Agent Doesn't Need an LLM for Every Decision
Decision models turn bounded questions into typed probabilities. Where they fit in an agent loop, why calibration matters, and what Clef and Jev change.
On this page 6 sections
An agent is about to call a tool. It needs to answer one question: should this action proceed?
You can ask its LLM. The model generates an answer, your code reads it, and the workflow branches. Repeat that at every step, across thousands of tasks, and a small decision becomes a recurring cost in latency, tokens, and output handling.
Decision models offer a different contract: state in, typed probabilities out.
You supply the context and a schema of allowed answers. The model scores those answers in parallel, and your code decides what to do with the probabilities. There is no generated explanation to parse.
TypeSafe AI introduced Jev on September 15, 2026. On October 1, Cloudflare released Clef and Clef-flash on Workers AI, with Apache 2.0 weights and a Jev-compatible API. That gives us two implementations of an idea worth examining: an agent can use a dedicated decision model for bounded questions while its generative LLM handles the open-ended work.
What a decision model actually returns
A decision model answers bounded questions about a given state. The state might be a support ticket, an agent’s execution trace, or a domain. Clef also accepts images; Jev’s launch offering is text-based.
Each question declares its type and allowed answers. Here is the shape of Cloudflare’s support-ticket example:
{
"model": "clef",
"state": "Checkout has been failing for every customer for the last hour.",
"questions": {
"urgent": {
"type": "noul",
"instructions": "Is this support request urgent?"
},
"team": {
"type": "choice",
"instructions": "Which team should handle this request?",
"criteria": {
"billing": "Payments, invoices, and refunds",
"technical": "Outages, errors, and configuration",
"sales": "Plans and upgrades"
}
},
"severity": {
"type": "score",
"instructions": "How severe is the customer impact?",
"criteria": ["No impact", "Minor", "Major", "Critical"]
}
}
}noul returns a probability from 0 to 1 that a statement is true. choice selects among named categories, while score uses an ordered scale.
The response contains probabilities for the allowed answers. An illustrative result might assign 0.94 to urgent, 0.83 to technical, and 0.61 to Major severity. These are example values, not a measured response. Your code then applies policy: route to technical, escalate, or ask for review when the evidence is insufficient.
That contract differs from a conventional supervised classifier with a fixed output head. You can declare new categories at request time without retraining the model for each category set. It still needs evaluation on the new task; accepting a schema does not establish accuracy on it.
Generative LLMs can also use constrained decoding and structured-output APIs to produce valid JSON. The distinction here is the inference path: a decision model scores the declared alternatives directly, without generating an answer token by token.
The speed comes from skipping the decoding loop
Cloudflare describes Clef as a Qwen backbone followed by a specialized scoring head.
The backbone runs a prefill-only pass over the input. The head then scores valid schema choices in parallel. Cloudflare’s architecture uses option-specific evidence routing, attention between fields and back to the input, and schema-bound scoring. The output comes from internal model representations rather than generated text.
The decision step is non-autoregressive. It avoids the decoding loop that would otherwise write the answer one token at a time.
That removes one source of latency, but it does not make inference a fixed-cost operation. Longer inputs and larger schemas still require more computation. A yes/no gate and a twenty-option schema should be measured separately on the traffic you expect to serve.
Across its reported evaluation runs, Cloudflare gives these latency figures:
| Model | Median latency | p95 latency |
|---|---|---|
| Clef-flash | 38.8 ms | 122.4 ms |
| Clef | 209.3 ms | 238.6 ms |
| Jev | 524.1 ms | 536.0 ms |
Clef-flash’s median is about 13.5 times faster than Jev’s in that comparison. At p95, the gap is about 4.4 times. Both figures are useful, but production planning also needs your input sizes, network overhead, concurrency, and timeout behavior.
These are Cloudflare’s measurements, not an independent evaluation for this article.
A probability is useful only if you can trust it
The output format solves one problem: the model cannot invent a category outside the schema. It can still assign high probability to the wrong allowed answer.
For an agent, that distinction matters more than a clean JSON response.
Suppose you let an action proceed when its approval probability exceeds a threshold. If the model is overconfident, the gate can approve a bad action instead of deferring it. The number looks precise while the policy behaves badly.
This is why calibration matters. Across comparable predictions, an answer assigned 0.9 probability should be correct roughly 90% of the time. That property needs to hold on your workload, including the cases near your operating threshold.
Cloudflare reports training Clef’s routing head and low-rank adapters with label-smoothed cross-entropy and a Brier loss to improve calibration. Its Reinforcement Learning for Calibrated Decisions objective also gives partial credit for adjacent ordinal answers: predicting Major when the truth is Critical receives different treatment from predicting No impact.
Those training choices are encouraging. They do not establish calibration on your traffic.
Before using a threshold to grant more autonomy, measure false approvals, false rejections, and how often the workflow defers. Inspect performance by action type and risk level. A single aggregate accuracy score can hide the failures that cost the most.
Where it fits in an agent loop
The upper path generates text and then interprets it. The lower path scores the allowed answers and leaves the action to explicit policy. The latency figures shown are reported by Cloudflare.
The useful pattern is to put a decision model on recurring, bounded branches in the agent loop. The generative LLM still investigates the problem, writes the fix, or drafts the response.
| Decision point | The question | Why a decision model fits |
|---|---|---|
| Support ticket routing | Urgent? Which team? How severe? | High volume, declared categories, and thresholds for escalation |
| Tool-call gating | Should this proposed action proceed? | Repeated at every step, with different costs for false approvals and rejections |
| Domain classification | Which categories describe this website? | Cloudflare reports a fetch, render, and classify workflow taking 2.2 seconds |
| Trust and safety triage | Which policy applies? How severe is the case? | Bounded labels and historical decisions for evaluation and domain tuning |
| Human escalation | Does the evidence support an automated decision? | Policy can use the returned probabilities to defer uncertain cases |
Tool-call gating is the pattern I find most interesting, and one I use today: Jev sits in my agent loop as a yes/no gate before sensitive tool calls.
The asymmetry does the design work. A false rejection may cost a retry or a human review. A false approval may allow a harmful action. That gives me a reason to begin with a conservative threshold and change it only when observed results justify doing so.
The model’s score is one input to that gate. Tool permissions, explicit user authorization, and execution limits remain rules enforced by the runtime. High model confidence does not grant access the agent was never given.
Where it falls short
A decision model works within the answers you declare. If the correct answer is missing, it may confidently select an unsuitable option. Add an unknown or defer outcome where it makes sense, and test how the model behaves when the input falls outside the intended task.
It also does not generate a plan, an explanation, or a repair. A workflow can call it repeatedly to choose among actions, but it still needs other components to construct those actions and carry them out.
Model selection remains workload-specific. In Cloudflare’s published benchmark table, Jev leads Clef and Clef-flash on When2Call and BRIGHT. A lower latency figure does not make Clef-flash the best choice for every gate.
API compatibility helps with experimentation. Cloudflare says Clef is Jev-compatible, which can reduce integration work. Switching providers still calls for checking authentication, endpoint configuration, supported inputs, and the response contract. Existing policy thresholds need re-evaluation because a new model’s probability distribution may behave differently.
I would keep the model behind an interface, evaluate it on representative decisions, and monitor it for drift. That makes a provider change an engineering choice supported by evidence.
What the competition changes
Compatible APIs make it easier to compare decision models without rebuilding the workflow around each one.
The work around the model becomes more valuable: defining useful questions, choosing complete answer sets, collecting reliable labels, measuring calibration, and setting policy around the consequences of being wrong.
Cloudflare is also offering hands-on fine-tuning through its forward-deployed engineering team. Its announced self-service platform is a future step. The proposed pipeline combines captured requests in AI Gateway, rollouts on Workers AI, scoring in Containers, training, and redeployment.
Historical decisions can help with domain tuning, but the labels need scrutiny. A log of past approvals records what the system allowed. It becomes evidence of correctness only when you connect those approvals to reliable outcomes.
In The Harness Decides When the Work Is Done, I argued that completion should be an explicit runtime contract. Decision models bring a similar discipline to the branches inside a task: declare the question, bound the answers, measure the probabilities, and let policy determine the action.
The next improvement to an agent may be a better interface for a small decision it makes thousands of times a day.
Sources and further reading: Cloudflare, Introducing Clef: our open-source decision models, and new RL fine-tuning platform, including the Clef API documentation, Clef weights, and live decision-index demo; TypeSafe AI, Introducing System One Models & Jev; and the Jev Decision Index. Latency and benchmark figures are reported by Cloudflare and have not been independently verified for this article. The agent-loop framing, policy recommendations, and tool-gating experience are my own.