AI agent / LLM-powered service

🤖 AI Agent → Production live

Run an AI agent through 5 gates: spec → prototype loop → MVP with evals → safety & cost → production

▸ 01 Stage 1 · Agent spec: job, guardrails, eval set live

Define the agent's job so narrowly that success is measurable: inputs, tools, refusal policy, and a golden eval set written BEFORE the first prompt.

Gates

  • The agent's job is one sentence with a measurable success criterion
  • A golden eval set (≥ 20 cases) exists, written before any prompting
  • Refusal policy is explicit: what the agent must NOT do or answer
  • Tool list is minimal and each tool has a purpose

Catalog skills

Prompts

Prompt: 01-idea-to-spec~177 words

# Idea → One-page Spec

Turn the idea below into a one-page spec a single engineer can start tomorrow.

When to use

  • Stage 1 of any conveyor, before writing a single line of code
  • Any time the scope of an ongoing project quietly grows

Prompt

You are a product engineer. Convert the idea below into a one-page spec.
Rules:
- Cut scope aggressively: exactly ONE runnable vertical slice.
- No feature without a measurable success metric. Unmeasurable wishes go to Non-goals.
- The slice must have a Definition of Done checkable in one working day.
- Output exactly these sections, nothing else:
  1. Problem (2 sentences, who hurts and why)
  2. Target user (one persona)
  3. Success metric (one number with a deadline)
  4. The one slice (steps a user takes, first to last)
  5. Non-goals (at least 3 explicit rejections)
  6. Definition of done (checkbox list)

Idea: <PASTE YOUR IDEA>

Gate this satisfies

Stage 1 · gates 1-4: spec exists, scope cut to one slice, non-goals written, DoD checkable.

Prompt: 06-agent-eval-set~165 words

# Golden Eval Set Before Any Prompting

Write the agent's golden eval set BEFORE tuning a single prompt. No eval set = no objective quality, only vibes.

When to use

  • Stage 1 of the AI conveyor, before writing the first prompt
  • Every time the agent's job changes scope

Prompt

You are an AI QA engineer. Build a golden eval set for the agent below:
- >= 20 cases across 5 categories: happy paths (8), edge inputs (4),
  refusal cases (4), adversarial/injection (2), regression anchors (2+)
- Each case: input, expected behaviour (observable, not vibes), severity
- 3 cases MUST be ones the current design will fail (they define the next work)
- A runner spec: how to execute the set and compute a pass rate
Output: the eval set as a table + the runner spec.
Agent job: <ONE SENTENCE + TOOLS>

Gate this satisfies

Stage 1 · gates 2-3: eval set written before prompting, refusal policy explicit.

Artifacts

  • Agent spec one-pager + refusal policy
  • Golden eval set (cases + expected outcomes)

▸ 02 Stage 2 · Prototype the agent loop live

The smallest working loop: prompt → model → tool → response, run locally against the eval set. Judge with the eval set, not with vibes.

Gates

  • The loop runs locally: prompt → model → tools → structured response
  • The golden eval set runs and prints a pass rate
  • Failures are categorized (bad prompt / bad tool / bad data)
  • Model choice is recorded with cost per eval run

Catalog skills

Prompts

Prompt: 02-skeleton-and-smoke~164 words

# Skeleton + One-command Smoke Test

Generate a throwaway prototype: the smallest project that runs the slice locally with one command and proves it with a smoke script.

When to use

  • Stage 2 of any conveyor: the prototype loop
  • Deciding whether an idea lives or dies before investing

Prompt

You are a pragmatic engineer. Create a THROWAWAY prototype (ugly code is fine) for the spec below.
Requirements:
- One command boots everything locally: `docker compose up` (or equivalent)
- One happy path works end-to-end
- A smoke script `scripts/smoke.sh` exits 0 when the slice works, 1 when it does not
- No auth, no polish, no extra features — the slice only
- A DECISION.md file: "continue / pivot / kill" plus the reason

Output: the full file tree with file contents, then the two commands to run.
Spec: <PASTE THE SPEC>

Gate this satisfies

Stage 2 · gates 1-3: one-command local run, E2E smoke, recorded decision.

Prompt: 06-agent-eval-set~165 words

# Golden Eval Set Before Any Prompting

Write the agent's golden eval set BEFORE tuning a single prompt. No eval set = no objective quality, only vibes.

When to use

  • Stage 1 of the AI conveyor, before writing the first prompt
  • Every time the agent's job changes scope

Prompt

You are an AI QA engineer. Build a golden eval set for the agent below:
- >= 20 cases across 5 categories: happy paths (8), edge inputs (4),
  refusal cases (4), adversarial/injection (2), regression anchors (2+)
- Each case: input, expected behaviour (observable, not vibes), severity
- 3 cases MUST be ones the current design will fail (they define the next work)
- A runner spec: how to execute the set and compute a pass rate
Output: the eval set as a table + the runner spec.
Agent job: <ONE SENTENCE + TOOLS>

Gate this satisfies

Stage 1 · gates 2-3: eval set written before prompting, refusal policy explicit.

Artifacts

  • Eval runner with a printed pass rate
  • Prototype loop code (explicitly throwaway)

▸ 03 Stage 3 · MVP: structured output + RAG + deployed live

The agent becomes a service: schema-constrained outputs, grounded answers (RAG or tools), CI that blocks on eval regressions, and a public endpoint.

Gates

  • Outputs are schema-constrained (no free-text parsing of model answers)
  • Grounding is real: every factual answer cites its source
  • CI blocks on eval pass-rate regression
  • A public endpoint serves the agent behind HTTPS

Catalog skills

Prompts

Prompt: 03-mvp-cut-and-ci~170 words

# MVP Scope + CI Pipeline

Upgrade the prototype into a deployed MVP: real data model, auth, CI gates, public URL.

When to use

  • Stage 3 of any conveyor, when the prototype decision was "continue"
  • Any project that runs manually but should run in CI

Prompt

You are a senior engineer. Upgrade this prototype into an MVP with these gates:
1. Real data model with migrations (no hand-edited databases)
2. Auth: working session or token flow; secrets in a manager, never in code
3. CI pipeline on every push: lint -> tests -> build -> deploy; red blocks merge
4. A public HTTPS URL serving the MVP, plus a documented ROLLBACK command
Keep the prototype's scope. Add NOTHING that is not in the spec.
Output: a numbered task list (each task < 2h), the CI file, and the deploy config.
Prototype: <PASTE THE REPO TREE>

Gate this satisfies

Stage 3 · gates 1-4: CI blocks red, migrations, auth + managed secrets, public URL.

Prompt: 06-agent-eval-set~165 words

# Golden Eval Set Before Any Prompting

Write the agent's golden eval set BEFORE tuning a single prompt. No eval set = no objective quality, only vibes.

When to use

  • Stage 1 of the AI conveyor, before writing the first prompt
  • Every time the agent's job changes scope

Prompt

You are an AI QA engineer. Build a golden eval set for the agent below:
- >= 20 cases across 5 categories: happy paths (8), edge inputs (4),
  refusal cases (4), adversarial/injection (2), regression anchors (2+)
- Each case: input, expected behaviour (observable, not vibes), severity
- 3 cases MUST be ones the current design will fail (they define the next work)
- A runner spec: how to execute the set and compute a pass rate
Output: the eval set as a table + the runner spec.
Agent job: <ONE SENTENCE + TOOLS>

Gate this satisfies

Stage 1 · gates 2-3: eval set written before prompting, refusal policy explicit.

Artifacts

  • Eval CI job (blocks merge on regression)
  • Deployed agent endpoint

▸ 04 Stage 4 · Safety & cost: guardrails, caching, observability live

Assume users will try to break it and finance will ask what it costs: prompt-injection defenses, refusal enforcement, per-request cost/latency tracking, and caching where it pays.

Gates

  • Prompt-injection test suite exists and passes
  • Refusals are enforced in code, not only promised in the prompt
  • Cost and latency per request are logged and alertable
  • Caching measurably reduces cost without hurting the eval pass rate

Catalog skills

Prompts

Prompt: 07-guardrails-and-cost~149 words

# Guardrails, Refusal Enforcement, Cost Budget

Assume users will attack the agent and finance will audit its bill. Produce the safety-and-cost plan.

When to use

  • Stage 4 of the AI conveyor, when the agent MVP is public

Prompt

You are an AI platform engineer. Harden this agent:
1. Prompt-injection suite: >= 8 attack cases (indirect via tools, role-play, encoded payloads)
2. Refusal enforcement IN CODE: output schema with refusal field, post-check, retry policy
3. Cost/latency: per-request accounting, monthly budget, alert at 60% and 90%
4. Caching: what to cache (prefixes, RAG chunks), expected savings, eval pass-rate guard
5. Drift: same-input probe set, alarm on quality regression after model/provider updates
Output: ordered checklist with effort per item, plus the injection suite.
Agent: <PASTE THE LOOP AND STACK>

Gate this satisfies

Stage 4 · gates 1-4: injection suite passes, code-enforced refusals, cost/latency alerting, measured caching.

Artifacts

  • Prompt-injection test suite
  • Cost/latency dashboard with alerts

▸ 05 Stage 5 · Production: evals in CI, monitoring, drift control live

The agent is operated like a service, not demoed like a chat: evals on every change, model/provider drift monitored, incidents postmortemed into the eval set.

Gates

  • Every prompt/model change runs the eval set in CI before merge
  • Production responses are sampled and re-scored against the eval set
  • Model/provider drift has an alarm (same input, worse output)
  • Every incident becomes a new eval case

Catalog skills

Prompts

Prompt: 06-agent-eval-set~165 words

# Golden Eval Set Before Any Prompting

Write the agent's golden eval set BEFORE tuning a single prompt. No eval set = no objective quality, only vibes.

When to use

  • Stage 1 of the AI conveyor, before writing the first prompt
  • Every time the agent's job changes scope

Prompt

You are an AI QA engineer. Build a golden eval set for the agent below:
- >= 20 cases across 5 categories: happy paths (8), edge inputs (4),
  refusal cases (4), adversarial/injection (2), regression anchors (2+)
- Each case: input, expected behaviour (observable, not vibes), severity
- 3 cases MUST be ones the current design will fail (they define the next work)
- A runner spec: how to execute the set and compute a pass rate
Output: the eval set as a table + the runner spec.
Agent job: <ONE SENTENCE + TOOLS>

Gate this satisfies

Stage 1 · gates 2-3: eval set written before prompting, refusal policy explicit.

Prompt: 07-guardrails-and-cost~149 words

# Guardrails, Refusal Enforcement, Cost Budget

Assume users will attack the agent and finance will audit its bill. Produce the safety-and-cost plan.

When to use

  • Stage 4 of the AI conveyor, when the agent MVP is public

Prompt

You are an AI platform engineer. Harden this agent:
1. Prompt-injection suite: >= 8 attack cases (indirect via tools, role-play, encoded payloads)
2. Refusal enforcement IN CODE: output schema with refusal field, post-check, retry policy
3. Cost/latency: per-request accounting, monthly budget, alert at 60% and 90%
4. Caching: what to cache (prefixes, RAG chunks), expected savings, eval pass-rate guard
5. Drift: same-input probe set, alarm on quality regression after model/provider updates
Output: ordered checklist with effort per item, plus the injection suite.
Agent: <PASTE THE LOOP AND STACK>

Gate this satisfies

Stage 4 · gates 1-4: injection suite passes, code-enforced refusals, cost/latency alerting, measured caching.

Artifacts

  • Eval-in-CI job on the main branch
  • Production sampling dashboard + drift alarm

Source: conveyors/ai-agent-to-production.json · prompts