← Nitish Gupta — AI Product Manager

The harness — how one person ships products a model builds

One SOP, four role-scoped review agents, three hooks and a 60 KB instruction budget. On one build: 45 phase specs and 1,711 tests for USD 661.57 of metered model spend. The same SOP, carried unchanged to a second product.

1 · Context and constraint

One person, two to three products in flight, and a coding model as the primary implementer. The person is architect, reviewer and approver; the model drafts specs, writes code and runs tests.

2 · Problem

Two failure modes, both measured.

Cost. One verification stage on the FrameProof build processed about 150 M tokens in 33 minutes, 132 M of it in nine sub-agents, because 468 KB of "rules" were being loaded into every agent on every call (Agents on a budget). The project had a documented cost policy; nobody had measured sub-agent spend.

Instructions that do not bind. A rule written in prose is obeyed until it is not. Permission patterns looked like the mechanical answer, but they are prefix matches on the whole command: on the video project, 2,332 of 2,389 recorded shell calls began with a change-directory step, so a rule written to catch a risky command matched almost nothing (measured from the guard hook's own header).

The question this page answers: what does a delivery system look like when the builder is a model, the reviewer is one person, and neither cost nor scope can be left to the model's good intentions?

3 · My role

Author of the delivery SOP, every project's rules file, the review agents and the hooks. Operator on every phase: approver of each spec, reviewer of each build, sole signatory on each sign-off. The model never signs: the SOP reserves complete for the operator's explicit approval.

4 · Decision and trade-off

The decision: treat the instructions an agent follows as production configuration — budgeted, reviewed and, where it matters, enforced by something that runs whether or not the model read them. Four calls follow from it.

1. Rules as small auto-loaded files with a hard budget, not a longer, smarter prompt. A rule the session never loads is not followed; a rule loaded everywhere is paid for on every call (133× mean amplification, measured). The answer was a 60 KB budget per project and a session-start hook that warns when it is crossed. Rejected: one comprehensive instruction file, the state the cost incident started from.

2. Enforcement climbs a ladder, and the rung is chosen by what a miss costs. The honest history is that the ladder was climbed one rung at a time:

flowchart LR
  P["Prose rule
cheap, can be skipped"] -->|"ignored or misread
once"| T["Tool allow-list
per agent"]
  T -->|"scope still stated
only in prose"| H["Hook on every tool call
cannot be skipped"]

On the OSINT build, the four review agents had tool allow-lists but no deny-lists: each one's write scope (one folder, one file, or a log only) was stated in its prose. On the later projects the scope moved into mechanism: the two cold-start reviewers in the prep workflow are given one tool, Read; the quiz project's boundary hook denies access to private folders for file tools and shell commands, because a file-tool deny rule does not cover a shell cat. Trade-off accepted: a hook asks the person more often than a rule would. That is the price of a check that cannot be skipped.

3. Reviewers may not fix. Every review is role-shaped (a spec gets product fitment and testability; an implementation gets code review), the outcome is binary — approved or revise — and the reviewer writes its findings, never the fix; the orchestrating session records the resolution. Rejected: an auto-fixing reviewer, which would review its own repair.

4. Evaluation floors as code, pinned before the build. The floor is a decision trigger, "reported, never quietly absorbed" (SOP III.2). On the OSINT build the gate script encodes an 85.9 % band floor and blocked a proposal that would have cost 6.6 %. Rejected: review by reading, which agrees with whatever the reviewer expected.

5 · What shipped

The loop every project now runs:

flowchart LR
  G["Three session gates
budget · scope shape · model"] --> S["Phase spec
NFRs declared, eval floor pinned"]
  S --> R1{"Role-shaped review
approved / revise"}
  R1 -->|revise| S
  R1 -->|approved| B["Build, AI-assisted"]
  B --> R2{"Code review
reviewer may not edit"}
  R2 -->|defects| B
  R2 --> E{"Eval gate in CI"}
  E -->|below floor| B
  E -->|at or above| O["Operator: live demo,
explicit approval, sign-off"]

• Problem: the operator re-explained process every session. Action: wrote a delivery SOP that answers process questions before they are asked — three gates, a phase-spec skeleton, a three-condition sign-off, a memory contract. Result: the same file, byte-identical, governs a second product (MyWarren).
• Problem: one reviewer, everything reviewed once. Action: four role-scoped review agents on the OSINT build — spec drafter, test lead, code reviewer, end-user subject-matter expert — each with its own framing and a narrow write scope. Result: 45 phase specs and 29 phase sign-off tags on that build.
• Problem: rules that cost tokens on every call. Action: the 60 KB budget and a payload-size hook, ported to each new project. Result: first-call context 256,372 → 59,730 tokens (−77 %) on the incident project; MyWarren's auto-loaded payload 135.4 KB → 59,099 B.
• Problem: permission patterns that matched nothing. Action: a guard hook that splits a shell command into segments and inspects each, and a boundary hook for private folders. Result: the risky token is caught wherever it sits in the command. The hooks do not log their decisions today, so there is no catch count to quote.
• Problem: a capped progress log that could lose history on roll-off. Action: a roll-off script that refuses to write unless every block is byte-identical and in order across both files. Result: fifty-plus sessions of state carried across context resets with nothing summarised.

6 · Measured result

What is not measured. There is no control: no build of the same scope without the harness, so nothing here proves the harness caused the throughput. Hook catch rates are not logged. The OSINT build's root-cause-analysis and code-review counts are not quoted because two sources disagree and have not been reconciled.

7 · What was deliberately not built

• An agent framework. LangChain was in the submitted stack and not adopted; the loop is a plain tool-using workflow whose control flow the code decides. Flip: a second team building on the same orchestration.
• Hosted tracing and evaluation. Excluded on the air gap; evaluation lives in a gate script in the repository. Flip: a deployment that permits egress and needs traces across more than one operator.
• An auto-fixing reviewer or hook. Reviewers report; the orchestrating session fixes. Flip: none foreseen. A reviewer that repairs its own finding has stopped being a reviewer.
• A hook in every project. Hooks live where an incident happened; other projects carry the 60 KB rule as prose in the user-level configuration. Flip: a second overrun in a project without one.
• Deny-lists on the OSINT build's review agents. Not deliberate in the first instance: their write scopes were prose. It is named here because the later projects moved scope into tool lists and hooks. Flip, already taken on later projects: any scope whose breach would touch private or submitted material. Whether any review agent wrote outside its stated scope on that build has not been checked.

8 · Assets

Downloadable skeletons (PRD outline, evaluation framework outline, sign-off log form, the SOP's phase-gate diagram, a case-study template) and further How I build pages are not published yet. Neither is code for the hooks.