How I build · The artefact set
The one idea. A PM artefact library for an AI product has two documents a conventional library does not: an evaluation framework that is a first-class artefact, kept separate from the KPIs because it answers a different question; and a risk register that maps each LLM-specific risk to a published taxonomy rather than inventing one. Everything else is the familiar set.
Reading path. Entry from The harness, the case study. The gates these artefacts feed are on a later How I build page, The delivery method; what changed between the planned and the built versions of the architecture and data model is on another, Plan versus as built. The project most of this was built for is a sovereign OSINT platform built for a defence innovation challenge.
1 · The library, by the decision each artefact makes
An artefact that does not make a decision is a report. Each line below names the decision the document exists to make, and, where it can be shown, what is different about it when the product has a model in the loop. Where nothing is claimed, the row says so.
| Group | Artefact | The decision it exists to make | What is different when the product is AI |
|---|---|---|---|
| Discover | Proto-personas, plus a negative persona | Who the product is for, and who it must not serve | The negative persona is the scope device: the user the system should refuse or abstain for |
| PRD with NFRs | What MVP 1 does, what it binds itself not to do, and how each story is accepted | Non-functional requirements carry latency, abstention and citation targets, declared before the build and measured before sign-off | |
| Measure | Success metrics: north star, KPI tree, guardrails | Whether the product is working for its users | A guardrail metric is one that must not degrade while a KPI improves (see §2) |
| Eval framework | Whether the model's behaviour is acceptable enough to sign off | A separate document from the metrics, with a floor encoded in a gate script (§2) | |
| Govern | Risk register | Which risks are accepted, mitigated or declined | No AI-specific field claimed; the AI-specific part is the mapping below |
| AI governance register | Which controls exist for data, model and output | The register of controls, kept apart from the register of risks | |
| LLM-risk mapping | Which published risk category each LLM-specific risk belongs to | Maps to the OWASP Top 10 for LLM Applications and to the NIST AI Risk Management Framework functions, so a reviewer can audit coverage against a list they already know | |
| Build-vs-buy record | What was adopted, what was declined, and why | Declined options carry their reason: a licence, the air gap, or a fixed control flow (§3) | |
| Plan | MVP roadmap | What ships in which increment, and what is deliberately deferred | No AI-specific field claimed |
| Grouped backlog | What is next, by theme | 111 entries in nineteen group files rather than one list, so an increment can take a whole group | |
| Model | Conceptual data model (CDM) | What the entities are and how they relate, in the language of the user | Has a planned and an as-built version (Plan versus as built) |
| Logical data model (LDM) | How the entities are stored and keyed | Models the vector leg and the full-text leg that retrieval fuses into one ranked list | |
| Design | Architecture; data-flow diagram (DFD); high-level and technical design (HLD, TDD) | How the parts connect and where the trust boundaries are | The architecture has a planned and an as-built version; the difference is the subject of Plan versus as built |
| Operate | Demo runbook | How the system is shown, on which data, in which order | No AI-specific field claimed |
| User guides | How a user operates the system | No AI-specific field claimed | |
| Method | Delivery SOP; sign-off log; RCA; retrospective | Who builds, who reviews, who signs, and what is learnt when the build drifts | The model is the builder and the human the reviewer and signatory (The delivery method) |
Where each artefact's output goes:
2 · Why the eval framework is not the metrics document
A KPI measures what the product does for its users: time to a cited answer, briefs produced per analyst, sources covered. An eval measures what the model does when asked: does it cite, does it abstain when the corpus has no answer, does it stay inside its scope. The two move independently. A model can score well on evals while the product fails its KPIs, and a product can hit its KPIs while the model has quietly started to answer questions it should refuse.
So they are two documents, read by two reviews:
What it bought, measured. On the OSINT build the eval floor for the credibility engine's band agreement was pinned at 85.9 % and encoded in a gate script. The gate later blocked a proposed change costing 6.6 %. The generation layer's sign-off gate required citation validity of 100 % (43 of 43) and abstention on 12 of 12 unanswerable questions; a smaller candidate model was rejected on that gate (abstention 9 of 12), not on a KPI.
What is in the eval framework and not in the metrics. The definition of eval for this product and why it is separate; an audit of the current state before anything is adopted; the principles; the candidate open-source frameworks considered; an adoption roadmap by MVP; the one near-term action for MVP 1; the guardrail cases added; and how the document is maintained. That is the outline offered as a download in §4.
3 · A risk register with an LLM-specific mapping
A conventional risk register lists risks the team thought of. For a product with a model in the loop that is not enough, because the failure modes are new to most reviewers and easy to leave out. The register therefore carries a mapping: each LLM-specific risk is tagged with its category in the OWASP Top 10 for LLM Applications and with the NIST AI Risk Management Framework function it falls under (Govern, Map, Measure, Manage). A reviewer can then ask the only question that matters about coverage: which category has no entry?
Beside the register, a build-vs-buy record keeps what was declined, with the reason, because a declined option is a risk decision too. Three from the build:
| Declined | Reason recorded |
|---|---|
| An object-detection model under a copyleft licence | The licence reaches the delivered platform |
| Hosted model-tracing services | The target is air-gapped; nothing may call out |
| An orchestration framework that was in the submitted stack | Not adopted; the build keeps model-decided control flow out, by decision (the OSINT platform) |
4 · The skeletons, as downloads
Five skeletons are published under Resources: headings and field lists only, re-typed, with no content from any project. They are the shape of the documents, offered so a reader can judge the method rather than take the description on trust. Each opens as a page, with the plain Markdown file to download.
| Download | What it is | Fields or headings |
|---|---|---|
| PRD outline | The nine-section PRD used for an MVP | Problem statement · Scope and non-goals (binding) · Actors · User stories with acceptance criteria, including candidate stories surfaced, not committed · Functional requirements · Non-functional requirements · Metrics hook · Open questions and risks · Traceability appendix |
| Eval framework outline | The eight-section eval framework, kept separate from the metrics | What "eval" means here and why it is a separate document · Current state (audit) · Principles · Candidate frameworks considered · Adoption roadmap by MVP · Near-term action for the current MVP · Guardrail additions · Maintenance |
| Sign-off log form | One row per built item, three things before it closes | Item · Spec · End-to-end test against the live system (date, result) · Demonstration with real data (date) · Approval (role, date) · Status: built, awaiting sign-off or signed off · What is outstanding. Sections: Open · Recently closed · Scope-only · Hygiene flags |
| SOP phase-gate diagram | The delivery loop as a Mermaid source | Master spec → phase spec → spec review → build → code review → eval gate → sign-off → retrospective; RCA on drift |
| Case-study template | The eight-field template every case study on this site uses | Context and constraint · Problem · My role · Decision and trade-off · What shipped · Measured result · Deliberately not built · Assets |
The two documents a conventional library lacks, the eval framework and the LLM-risk mapping, are the reason the sign-off log has a column for an end-to-end test and the PRD has a non-goals section marked binding: a declined risk becomes a non-goal, a non-goal becomes an eval case, and the eval case gates the sign-off.
5 · Stated honestly
- The personas are proto-personas. They were written from domain knowledge and the challenge brief; no user interviews were run. The negative persona is a design device, not research.
- The SOP travelled; the library was not counted. The delivery SOP was carried byte-identical to a second build (the same 66,238-byte file). That build's artefact library has not been inventoried for this page, so no claim is made about it.
- Retrieval quality is unmeasured. The eval framework pins floors for citation validity, abstention and band agreement; it does not yet carry a retrieval-quality eval, and that is recorded as a known gap.
- The backlog figure is a count, not a claim. How many of the 111 entries shipped is on the sign-off log, not here.
Measured figures on this page
| Figure | Source |
|---|---|
| Band-agreement eval floor 85.9 %; gate blocked a change costing 6.6 % | the OSINT build's eval gate script |
| Citation validity 100 % (43/43); abstention 12/12; candidate model rejected at 9/12 | the OSINT build's generation-layer sign-off gate |
| 111 backlog entries across 19 group files | the OSINT build's backlog |
| SOP byte-identical across two builds, 66,238 bytes | byte comparison of the two SOP files, 2026-10-17; The harness |
What this page does not claim
- No artefact is reproduced. The table describes each document's purpose; the downloads carry headings and field names only. Programme and customer names appear inside the source files and appear nowhere here.
- No count of artefacts is quoted beyond the backlog entries. The source folder also holds interview-preparation notes, a cost note and requirements-trace files, which are not PM artefacts and are excluded by rule; a headline artefact count would include them.
- No field is claimed inside an artefact that was not read. Where the table says no AI-specific field claimed, that is the reason.
- No claim that the mapping is complete. The OWASP and NIST tags make gaps visible; they do not guarantee there are none.