UI Harness — a 20-node agent pipeline that ships production UI code
An agent pipeline that takes a written brief to built, validated Unity code, behind five human approval gates. It has shipped screens into a commercial mobile game.
| Scale | 20-node LangGraph pipeline · 10 role-specialised agents · 5 human gates |
| Code | ~34,200 lines of Python 3.12 · 1,018 tests |
| Workers | Claude · Codex · OpenHands · Unity MCP · deterministic |
| Output | Screens running in a released Unity game |
| Cost | $67–98 per accepted screen, attributed by cause |
What it does
- Turns a written brief into a typed contract, then into working Unity C# and art.
- Runs every artifact through 14 validator modules before a human sees it.
- Scores the result with a six-axis visual judge, then asks a human to approve or reject.
- Routes a rejection back to the stage that caused it, not to the start.
- Records token, tool-call, duration, cache, retry and cost data for every stage.
- Exposes the whole thing as an MCP server and a native desktop app.
Results
Three screens, taken end to end and accepted by a human reviewer. Every figure below is read from the pipeline's own kpi.json, written per run.
| Screen | Cost | Attempts | Tool calls | Cached tokens | Judge score |
|---|---|---|---|---|---|
hero-collection-v1 | $67.28 | 17 | 545 | 58.4M | 0.908 |
hero-collection-v2 | $97.73 | 27 | 591 | 62.7M | 0.913 |
library-screen-v3 | $78.64 | 34 | 597 | 46.9M | 0.642 |
These shipped. Both screen families landed in the game on 2026-08-18.
Where the money goes, on the run that measured it:
- First attempt: $34.14
- Reruns: $63.59 — 65% of the total
- Of that: $47.76 validator, $10.68 human, $5.15 crash
Knowing that reruns cost twice the first pass, and that three quarters of rerun spend is validator-driven, is what makes the next optimisation obvious instead of a guess.
What the stages produce


Both are artifacts of one recorded run, combat-screen-6643dafd0c.
How it works
Typed contracts are the only source of truth
Pydantic models are canonical. Markdown and HTML are generated views. A stage cannot hand the next stage prose it invented — it hands over a validated object or it fails.
Human gates, with rejection routed to the causal stage
Five gates: readiness, sketch, layout, art, final. A rejection at the art gate returns to art direction, not to the top of the graph. Rejections can be element-anchored:
reject --annotate "HeroCard: portrait is cropped"- The annotation is applied as a minimal JSON-Patch on the presented contract.
- Everything not annotated is preserved.
- The re-raised gate lists the exact delta.
An evaluation harness, built before the features
- 14 validator modules — geometry, components, compilation, naming, pixel scale, asset presence, policy, code rules, live Unity state, mockup comparison.
- Deterministic checks run before the visual model is called, so a cheap failure never costs an expensive review.
- Validator findings get a bounded JSON-Patch repair cycle before they cost a human turn.
- The visual judge scores six axes: composition, readability, style conformity, asset completeness, mockup alignment, defect-free.
Judge-vs-human agreement is measured
The pipeline records whether its automated judge and the human reached the same verdict:
| Run | Judge | Human | Agreement |
|---|---|---|---|
hero-collection-v1 | pass | approved | ✅ |
hero-collection-v2 | pass 0.913 | rejected | ❌ |
library-screen-v3 | fail 0.642 | approved | ❌ |
An evaluator you have not checked against a human is an assumption. This one is checked every run, and the disagreements are the input to the next round of prompt and rubric work.
Retrieval, scored against a gold set
- Hybrid dense + lexical over a local
sqlite-vecindex, fused by reciprocal rank. - Embeddings run locally through ONNX; the index is built incrementally, so unchanged chunks are never re-embedded.
- Scored against a 32-pair gold set built from real failure history, with queries deliberately not phrased in the target entries' wording.
- recall@10 0.97, MRR 0.66.
Cost control that never throws away paid work
- Per-stage model and reasoning-effort routing, with provider fallback chains.
- Budgets are advisory: exceeding one raises a warning, and completed valid output is kept. A budget that discards work you already paid for costs more than it saves.
- Usage is append-only per attempt, attributed to provider, requested model, effort, and the provider's own reported model breakdown.
Durability
- SQLite checkpointing; completed stages resume after interruption.
- Exclusive leases — one run writes to the Unity Editor at a time, one writer touches the harness's own source at a time.
- Stages are bounded on silence: five-second heartbeats, streamed provider events and process identity, so a hung worker cannot hold a run open indefinitely.
- Artifacts are immutable and content-addressed within a run.
The pipeline improves itself, and never applies its own changes
- Every accepted run generates a refinement proposal grounded in that run's KPIs.
- Proposals have a visible lifecycle: proposed → approved / rejected → applied.
- Applying is a human act. The harness records where the human landed the change and applies nothing itself.
replay — regression testing for prompts and validators
Re-judges every stored contract with the current validators against the human's recorded gate decisions. Reports regressions on previously approved contracts and new catches on rejected ones.
No Unity. No model calls. Prompt and rubric changes become checkable for free.
Problems I found and fixed
The instrumentation exists to catch things. These are three it caught.
Budgets set from intent are not controls
Found: every one of the 15 stage executions on the ledger had breached its output budget. readiness was set to 4,000 against observed 4,115 → 8,344 with a 17,284 outlier. art_direction was set to 10,000 against 20,593.
Why it mattered: a limit that everything always breaks is noise. It hides the one genuine blow-up you needed to see, and it trains the operator to ignore the only channel that would report it.
Fixed: re-tuned from the ledger at roughly p100 × 1.2. Every breach since is a real signal.
A fallback chain that had never been observed failing over
Found: the config advertised three providers. When Claude failed, the recorded fallthrough read codex: Executable probe failed with exit code 1; openhands: … — one provider failure was terminal, while the message implied three had been tried.
Root cause: the probed executable was the npm .cmd shim, which resolves node from PATH. The operator service's environment has no node, so cmd.exe exits 1 — and the identical command succeeds in an interactive shell, which is why it passed every manual test.
Fixed, in three parts:
- Discovery prefers the vendored native binary, which needs neither node nor
cmd.exe. - Probe failures carry the process's stderr tail instead of a bare exit code.
- Intake writes a
provider_availabilityevent, so every run states up front which fallbacks it actually has.
The fix that mattered was not the discovery glob. It was making availability an event on the ledger.
The system's memory worked against it
Found: archiving a resolved defect removed it from the pipeline's retrieved context — so the record of a solved problem stopped being available as precedent at the moment it became most useful.
Fixed: the feedback log is retrieved rather than pasted, chunked one entry at a time, and triaged rather than tidied. Resolved entries stay reachable. This is why retrieval exists in the system at all.
What the stages report about themselves
Every participant — orchestrator, readiness, technical designer, art director, code adapter, Unity implementer, asset creator, visual reviewer — appends to a shared feedback log whenever an input, handoff or tool gets in its way.
236 entries so far. They are retrieved into later runs, so the pipeline's own observations about its weak points become context for the work that follows.
Runs remember
Gate rejections, design-owned blocking findings and recovery instructions from earlier runs of the same screen family reach the design and review stages as bounded screen_memory context, projected deterministically from the durable ledgers.
--no-memory runs a control, so the feature's effect can be measured rather than assumed.
The operator — a native desktop app
The pipeline is driven from a desktop application I built for it: Tauri 2 + Rust (~34,900 lines) with a React/TypeScript renderer (~10,000 lines).
- Rust owns the window, tray, single-instance behaviour, project selection, sidecar lifecycle and an allowlisted bridge. Python owns durable project and task state.
- Creates harness runs, inspects stages, workers, telemetry and context, previews image evidence, approves or rejects the human gates, routes repairs, cancels and recovers runs.
- Monitors pipeline state with no browser, no localhost page, no terminal and no model calls — state comes from the durable ledgers, not from asking a model what happened.
Stack
Orchestration LangGraph · Pydantic · SQLite checkpointing · MCP server · asyncio Providers Anthropic · OpenAI · OpenHands · per-stage model and effort routing · fallback chains Retrieval hybrid dense + lexical · ONNX embeddings · sqlite-vec · recall@k · MRR Evaluation 14 validator modules · six-axis visual judge · JSON-Patch repair · replay Operator Tauri 2 + Rust desktop app · React/TypeScript · Unity MCP Language Python 3.12 · 1,018 tests · pytest
Evidence
Every figure on this page is read from the repository or from the pipeline's own generated reports, re-measured 2026-08-23. Source access can be arranged during a hiring process.





