Skip to main content
Clearlead AI Consulting
All articles
Paul Ferguson

Decision models: AI built to decide, not to write

The flexibility of an LLM at the speed (and cost) of a classifier

Documents, speech bubbles, and text fragments converge towards four choices on a navy background, with one teal choice marked with a tick.

Over the past few years, quite a lot of my work with large language models (LLMs) hasn't involved generating text at all: it has involved classification and other structured tasks, where the model reads something and makes a small judgement about it, such as which category a document belongs to, or whether a message needs a person to look at it. I wrote about this side of LLMs back in June 2024, in an article on their analytical capabilities (covering things like classification, sentiment analysis and topic modelling). I've also spent a significant part of my career building these things the more traditional way, with machine learning models trained for one specific task, so I have a fair appreciation of how much effort goes into getting them right.

That's a large part of why I'm so impressed by a new wave of models built specifically for these small judgements. The LLM approach I wrote about in 2024 worked well, but the trade-off for the flexibility they provide is that they are slower (at inference) and cost more than traditional modelling approaches. These new models provide that same level of flexibility (you still describe what you want in plain language), but because they're designed for this kind of task, they can do it far faster and more cheaply (and often at the same level of accuracy as much larger models). That opens up some genuinely new uses, as well as changing how older systems could be built, and for me it makes them one of the most exciting developments in AI right now (and given all the hype, and how quickly everything is moving, I don't say that lightly).

I'll admit that they aren't particularly flashy, but small decisions like these are basically what powers a large number of systems and workflows (what gets routed where, what gets flagged, what's allowed to go ahead), so making them fast, cheap and easy to change opens up a lot of possibilities.

How we used to build these tasks

To give a bit of context here, if you'd wanted to build a very standard classification model, such as a sentiment classifier ten or fifteen years ago (one that reads a product review and decides whether it's positive or negative), the usual route was supervised learning: you need to collect reviews, have people label them, prepare the text, train a model, then test it on reviews it hadn't seen. Topic modelling is another example of a traditional modelling process, which can find themes in documents without any labels, but you still needed a decent body of documents, a fitted model, and someone to work out what each theme meant: having done this many times in many different domains, I can tell you first-hand that it can be a tedious and iterative task.

LLMs changed that process a lot: instead of collecting labelled examples, you could describe the judgement you wanted in plain language, pass in the text, and have a reasonable first answer (without any model training involved), and when the criteria changed, you changed the description rather than retraining anything. That flexibility is what makes them so useful for a variety of different classification tasks.

The drawback is that you're using a large, general-purpose model, generating its answer one word at a time, to produce what is often a single label, and a well-trained task-specific model could often still do better on accuracy. At low volumes that's a perfectly sensible trade-off, but it's slow and fairly expensive to do millions of times, so you ended up choosing between flexibility on one side and speed, cost and accuracy on the other (Figure 1).

Chart with speed and cost per decision on the vertical axis and flexibility on the horizontal axis. A trained classifier sits top left (fast and cheap, but fixed labels), a general LLM sits bottom right (flexible, but slower and costlier), and a dashed arrow lifts it to a decision model at top right (flexible, fast and cheap).
Figure 1: Decision models remove the old trade-off between flexibility and speed.

What changes when a model is built to decide

Over the past month or so, several companies have released what I'll call decision models: models designed to return a bounded judgement rather than free text. You give it some evidence (a message, a document, a database row) and a question with a fixed set of possible answers, such as a support message plus the options Billing, Technical, Delivery or Other, and it returns an answer along with a probability for each option. It's a bit like the difference between answering an exam question in writing and ticking a box on a multiple-choice paper (Figure 2).

Two answer sheets for the question 'What is this ticket about?'. On the left, an open question with handwritten lines, labelled 'General LLM: writes an answer'. On the right, a multiple-choice sheet with Billing ticked from Billing, Technical, Delivery and Other, labelled 'Decision model: ticks a box'.
Figure 2: A general LLM writes its answer; a decision model picks from the options you give it.

A general LLM can already be told to reply with a single label, so the output format isn't really what's new: the difference is in how the answer is computed. Cloudflare's Clef, for example, reads the evidence and question in a single pass through an existing open model, then scores every permitted answer in parallel, so there's no word-by-word generation at all. Other providers don't necessarily do it the same way, but the common idea is to reuse a broad model's understanding of language with a decision step trained specifically for this kind of task on top: the flexibility of an LLM, with something much closer to the speed and focus of a specialised model.

It has all happened quite quickly. TypeSafe introduced Jev in mid-September, Cloudflare released Clef with open weights on 1 October, and OpenAI put its Decisions API into public beta five days later. Published input prices run from about $0.04 per million tokens (Jev and Clef-flash) to $0.24 (Clef), with nothing charged for output, so a million requests of around 2,000 tokens each would cost about $84 at Jev's listed rate. Cloudflare reports a median response time of 152 milliseconds for a request of roughly 800 tokens.

Ask a new question of the whole dataset

One of the simplest examples of what this enables comes from MotherDuck, a cloud data warehouse, which has added Jev as a function inside ordinary SQL queries. In its own benchmark, it asked Jev to sort 100,000 news articles into four topics:

  • Jev was 89% accurate, took 40 seconds and cost $0.50.
  • OpenAI's GPT-5.6 Terra was 88% accurate, took 32 minutes and cost $37.58.

So it is pretty clear that it can offer state-of-the-art performance on these sorts of tasks, for a fraction of the cost and time.

Sorting articles into these sorts of topics is a pretty standard classification task, but what this changes is that a non-technical analyst can work with this sort of data. When a question like this can be asked across a whole database table and you get the results back in under a minute for about fifty cents, you can treat it like any other query: try it, look at the results, reword it and run it again. Previously, a new question like that usually meant sampling a few hundred rows, waiting on a long batch job, or commissioning a classifier to answer one fixed question.

Take a support team's data as an example, with one row per ticket. Plenty of systems can already tag a ticket as a complaint, but that doesn't tell you what the customer is unhappy about, and two complaints can be about different things:

  • "The total on my invoice is wrong." (unhappy about the problem itself)
  • "This is the third time I've written about the wrong total on my invoice and nobody has fixed it." (unhappy that it still hasn't been dealt with)

The first is a billing problem; the second is a service problem, which is often a different team's job to fix. Working out which type each ticket is could be a fairly standard task for one of these models (where we provide not just the text of the ticket, but the full conversation, as well as the status of the ticket).

All of this can be embedded within SQL, to allow people to drill down into these issues, for example by counting each kind of complaint by product or by month.

AI functions in SQL aren't new, but being able to try a new question over everything, interactively and for a few cents, changes how you can explore text data.

Check what a system is about to do

Another area that could benefit from these decision models is guardrails: checks that can stop, redirect or escalate an input, an output or a proposed action before it goes any further.

Some of those checks are exact, and belong in standard software:

  • Does the user have permission to view a record?
  • Is a payment within a defined limit?
  • Is an email going to an approved domain?

Other checks need more complex interpretation:

  • Does this reply disclose something from the internal notes it was given?
  • Is this customer asking for an exception to the returns policy?
  • Does the action an agent is about to take actually match what the user asked for?

Agents are one obvious place for this, because they take actions rather than just produce text, but the same checks apply to any AI feature or workflow.

Using a model at every point where a check would help used to be too costly, so checks tended to be limited to one or two places. But with decision models it becomes realistic to run several narrow checks at each stage, which helps the system to be more robust and safer.

TypeSafe has provided some examples to show how this can work, checking every message and reply going to and from an AI assistant, where each message is put through a handful of yes/no questions (is it trying to override the assistant's instructions? does it ask for a diagnosis or a dosage? does it suggest the person may harm themselves?), and the model returns a probability for each. These are then compared to pre-defined thresholds, to decide whether the message passes, goes to a person for review, is blocked, or is routed to a support path (Figure 3).

A message fans out to three yes/no checks with example probabilities: trying to override the instructions (0.02), asking for a diagnosis or dosage (0.91), and suggests risk of self-harm (0.01). The results are compared with thresholds, which route the message to pass, human review, block or a support path.
Figure 3: Several narrow checks on each message, with thresholds deciding what happens next.

Check the work as it is produced

Guardrails ask whether something should be allowed to go ahead, but a related question is whether an output is actually right, and the same economics apply.

Verifying citations is another good use case, for example if a research paper contains a reference (along with a quote) to a different paper, there are two main things that need to be checked:

  • Check if the quote exists in the document, which can be done in standard software
  • A decision model can check whether the surrounding passage supports the claim, contradicts it, or does not address it at all

Another pattern is escalation, where a cheap "judge" handles the routine cases and passes the uncertain ones to a stronger (more expensive) model or a person.

React while the interaction is happening

The last kind of example is about timing, where a decision is only useful if it happens during an interaction rather than after it.

OpenAI's documentation for its GPT-Live voice product shows Decisions being called from the live transcript while someone is still speaking. In one example, a digital character picks its expression, gesture and intensity as the person talks; in another, spoken requests in a car are mapped to cabin controls, with anything broader handed off to a general model. These are integration examples rather than measured deployments, but they show the general pattern: many small, meaningful choices made in the moment, instead of one generated response at the end.

So what could you build?

Because so many systems and workflows rely on small decisions like these, being able to ask a question in plain language and get an answer back in a fraction of a second, for a fraction of a cent, opens up a lot of new possibilities. The kind of judgements we used to ration can now go wherever they're useful: across every record in a database, at every step of a workflow, or in the middle of a conversation.

For me, the introduction of these models has given me the opportunity to rethink what is possible in a few different types of projects, as there are so many applications that can be improved by having the ability to add additional small judgements that are almost instant and almost free. And I suspect many of the best uses haven't been thought of yet. It's also a good reason to go back over older use cases, from classifiers built around a fixed set of labels to LLM calls that only ever return a single word, as plenty of them could now be faster, cheaper, or able to answer a useful question that wasn't possible before.

I'm really looking forward to seeing what people build with these.

If you're thinking about where these models could fit into your own systems, or which of your existing use cases could benefit, get in touch.

References

Back to all articles

Let's Talk

Have a question about AI?

Book a free 30-minute call and we will answer it honestly.

Book a free call