Ultracode

Ultracode runs one hard task through an exhaustive multi-agent pipeline instead of a single pass: it decomposes the task, fans out several solver agents that attack it from different angles, has an adversarial critic try to break each candidate, then synthesizes the survivors into one answer. It's eVi's analogue of Claude Code's ultracode — more thorough and more trustworthy, at the cost of more model calls.

Overview

Why a fixed pipeline (not a model-authored script)

Claude Code's ultracode can lean on a strong model to write a bespoke orchestration script per task. eVi targets local models (qwen2.5-coder:14b and down) that can't reliably do that — so eVi's orchestration is fixed Python (evi/ultracode.py). The model is only ever asked to answer one concrete, role-scoped sub-prompt per stage (decompose / solve-one-angle / critique-one / synthesize) — the floor even a small model can clear. Each stage is a fresh headless agent, so per-stage context stays small no matter how long the pipeline.

How it works

1. Decompose — one agent maps the task into sub-goals + key risks (context for the solvers; no tools).
2. Solve (fan-out) — breadth solver agents run in parallel, each told to take a different angle (direct, first-principles, edge-cases, simplicity, performance, alt). Solvers get the mode toolset (default code).
3. Verify (adversarial) — a critic reviews each candidate for its single strongest flaw (or says APPROVE). Runs with no tools — a critic can't write files. With rounds > 1, each critique is fed back to its solver for a refine pass, then re-critiqued.
4. Synthesize (fan-in) — one agent merges the strengths and fixes the critiqued flaws into the final answer (keeping the best candidate verbatim if it can't improve it; ignoring any ERROR: candidate).

The fan-out reuses workflows.fan_out (the same concurrency primitive behind evi workflow's parallel blocks). The core is model-free — run_ultracode takes an injected run_one callable, exactly like evals.make_runners — so the CLI, REPL, and web each supply their own agent factory and the pipeline is fully unit-testable.

Local-model note: with a single local backend, inference serialises, so breadth buys a quality win (diverse angles + adversarial cross-check) more than wall-clock speed. Real parallel speedup needs a multi-GPU box or a federation peer.

Setup

Defaults live under [ultracode] in ~/.evi/config.toml:

[ultracode]
breadth = 3          # parallel solver angles (1 disables fan-out)
rounds = 1           # verify->refine cycles (0 skips critique — weakest-model escape hatch)
mode = "code"        # tool preset for solvers: chat | cowork | code
angles = []          # optional explicit angle names (empty = first `breadth`)
max_workers = 4      # cap on concurrent stage agents
auto_tune = true     # downshift breadth/rounds for tiny / short-context models
cheap_fanout = false # run the solver fan-out on [llm] fast_model (keep critic/synth on the main model)

auto_tune downshifts to breadth=2, rounds=0 for tiny models (size tokens 0.5b/1b/1.5b/3b or mini/small in the name) or short context (< 16k) so ultracode stays usable on weak backends.

Running ultracode on a smaller model

evi models recommend now also suggests a Fast model (the largest small/ fast companion that fits — e.g. qwen2.5:3b on a 16 GB GPU). Set it as the downshift/fast model:

evi models use qwen2.5:3b-instruct-q4_K_M --fast   # sets [llm] fast_model

fast_model is used by /fast and composes with ultracode: turn on fast mode (/fast on) and the whole pipeline runs on the small model — handy to keep the big model free, or to run ultracode fast on modest hardware. The bundled small options span qwen2.5:3b, llama3.2:3b, phi3.5:3.8b-mini, qwen2.5:1.5b, llama3.2:1b, and qwen2.5:0.5b.

Cheaper fan-out (per-stage model routing)

The N parallel solvers are the expensive part of a run; the single adversarial critic and synthesizer are where quality is won. So you can route just the fan-out to a cheaper model and keep the critic/synth sharp:

evi ultracode "<task>" --cheap-fanout                 # solvers -> [llm] fast_model
evi ultracode "<task>" --solver-model qwen2.5:3b      # solvers -> an explicit model
evi ultracode "<task>" --solver-model qwen2.5:3b --synth-model qwen2.5-coder:14b

Or make it the default in config:

[ultracode]
cheap_fanout = true   # needs [llm] fast_model set; no-op otherwise

Under the hood each stage is decompose | solve | verify | synthesize, and UltraConfig.stage_models maps a stage to a model id (empty = the main model) — the same per-stage routing the web POST /api/dispatch/ultracode honours via {cheap_fanout: true} or {solver_model: "<id>"}. With a single local backend this is a cost/VRAM lever (smaller model for the bulk of the calls) more than a speedup, since inference still serialises.

Usage

CLI

evi ultracode "refactor the auth module to remove duplicated token parsing, add tests"
evi ultracode "<task>" --breadth 4 --rounds 2 --mode code   # override config
evi ultracode "<task>" --json                               # full result incl. every stage

It prints each stage as it runs (> solve direct, > verify edge_cases, …) then the final answer.

REPL

/ultra <task>     run THIS turn through the pipeline
/ultra            toggle session-wide auto-ultracode on/off
/effort ultracode max reasoning + auto-pipeline every substantive turn (Claude-parity);
                  /effort high|medium|low|max clears it

With auto-ultracode on, substantive turns (non-trivial, non-greeting) run through the pipeline automatically; short/greeting turns fall through to a normal turn. The prompt shows an [ultracode] chip while it's active.

Web / Desktop

Settings → Ultracode: a task box with breadth/rounds/mode controls and a Run button. It shows each stage (collapsible) and the final answer. Backed by POST /api/dispatch/ultracode ({task, breadth?, rounds?, mode?} → {ok, answer, stages, config}). Like the eval/recipe runners it blocks until the whole pipeline finishes.

Examples

# A weakest-model run: one solver, no critique (~3 calls, barely more than `evi run`)
evi ultracode "summarise what changed in this file" --breadth 1 --rounds 0

# A thorough run: 4 angles, a refine round
evi ultracode "design a retry policy for the HTTP client" --breadth 4 --rounds 2

Notes / limits

• Cost: a default run is ~8 model calls (1 decompose + 3 solve + 3 verify + 1 synthesize); rounds=2 adds a refine + re-critique pass. Use --breadth 1 --rounds 0 to gauge cost cheaply first.
• Synthesis regression: a weak synthesizer can occasionally produce a worse answer than the best solver; the synthesis prompt mitigates this ("return the best candidate verbatim if you can't improve it"). Inspect stages with --json or the web panel if a result looks off.
• Conversation coherence: the /ultra and auto-ultracode paths store only the final answer in history (not the plan/angles/critiques), so /context under-reports the turn's real token cost.
• Stages that error return ERROR: … and are passed to synthesis (told to ignore them) — a failed stage never crashes the run.