Harness Engineering: The Scaffolding Around an LLM

Ask a language model a question through its raw API and you get back one thing: text. There's no memory of the last call, no ability to open a file, run a command, or check whether its own answer was correct. Every "agent" you've used — one that edits code, browses the web, or works through a multi-step task unattended — is a bare model wrapped in a much larger piece of software that most people never think about. That wrapper has a name: the harness.

Harness engineering is the discipline of building that wrapper well. It gets far less attention than prompt engineering or model choice, but for any agent that has to do real work over multiple steps, it's often the thing that determines whether the system is reliable or flaky — regardless of which model sits inside it.

What a harness actually is

A harness is the surrounding software system that decides what gets sent to the model on each turn, what tools the model is allowed to call, how the results of those tool calls get fed back in, and what ultimately happens with the model's output. The model itself is a function: context in, next tokens out. It has no opinions about when to stop, no way to actually run the code it writes, and no memory beyond what's in the context window you hand it. The harness supplies all of that structure from the outside.

Concretely, a harness typically owns four responsibilities. It constructs the context sent to the model each turn — assembling conversation history, system instructions, and the current set of tool definitions into a single prompt. It executes tool calls the model requests — running the actual file read, shell command, or API request, then formatting the result so it can be fed back in as new context. It decides when a task is done — recognizing a final answer, a stuck loop, or a turn budget that's been exhausted, and stopping accordingly. And it enforces permission boundaries — which actions run automatically, which need explicit human approval, and which are blocked outright regardless of what the model asks for.

The agentic loop

Underneath all of that, the harness's core mechanism is a loop: observe the current state and context, let the model decide on an action — respond directly, or call a tool — execute that action, fold the result back into the context, and repeat until the harness itself decides to stop. The model never runs this loop; it only ever answers the single question it's asked on a given turn. The loop, the state it carries forward, and the exit condition are all harness code.

def run_agent(task: str) -> str:
    context = build_initial_context(task, system_prompt, tool_specs)

    while True:
        response = model.generate(context)

        if response.is_final_answer:
            return response.text

        # The model asked for a tool — the harness decides what happens next
        if not is_allowed(response.tool_call):
            context = context.append(denial_message(response.tool_call))
            continue

        result = execute_tool(response.tool_call)
        context = context.append(response.tool_call, result)

        if turn_limit_reached(context) or task_looks_stuck(context):
            return escalate_to_human(context)

Notice how much of that snippet has nothing to do with the model at all: permission checks, turn limits, stuck-task detection, escalation to a human. A weak model wrapped in a careful loop like this can outperform a strong model wrapped in a naive one that just calls the API in a bare while True until it happens to stop producing tool calls.

Bare API call vs. harness

Calling a chat completion endpoint directly gives you exactly one request and one response. There's no persistent loop, no tool execution, and no state carried across turns unless you build it yourself — the model doesn't remember the previous call unless you resend the transcript, and it can't act on the world unless something outside it interprets its output as an instruction and carries it out. A harness is precisely that "something": the engineering layer that turns a stateless, single-shot model call into a stateful, tool-using agent that can work toward a goal across many turns.

Why harness quality can matter more than the model

Swap the same base model into two different harnesses and you can get two very different agents. A harness that keeps context tight — pruning stale history, summarizing long tool output, surfacing only the information relevant to the current step — helps the model pay attention to what matters. One that lets context balloon with irrelevant logs and dead-end attempts dilutes that attention and degrades every subsequent decision, even though the model "knows" the same things in both cases.

The same is true of tools. A harness that exposes a handful of well-scoped tools with precise, unambiguous descriptions gives the model a much better chance of picking the right one than a harness that exposes twenty overlapping tools with vague names. And a harness that retries a failed tool call sensibly — reading the error, trying a narrower approach, giving up gracefully after a bounded number of attempts — produces a far more reliable agent than one that either gives up immediately or retries the exact same failing call forever.

Permissions as a harness concern

The harness is also where safety boundaries actually live. A model can "request" to delete a file or push to a remote branch, but whether that request executes, prompts for approval, or gets rejected outright is decided entirely by the harness's permission logic — not by the model's judgment. Treating this as a harness responsibility, rather than something to solve purely through prompting, is what makes an agent's behavior predictable enough to trust with real, consequential actions.