How to Evaluate an AI Feature Before Writing Code

Start with the user decision, not the model

Describe the current workflow without using the word AI. Identify the person doing the work, the input available at that moment, the decision or artifact they need, and what happens next. Then name the bottleneck. Slow search, inconsistent classification, blank-page drafting, and risky approval are different problems even if a chat interface can demo all four.

Define a narrow claim: “suggest two cited answers for a support agent to review” is testable; “transform customer support” is not. Record the non-AI baseline—manual work, keyword search, rules, templates, or a form redesign—and the user outcome you expect to improve. Useful measures may include accepted suggestions, task completion time, correction rate, escalation quality, or successful resolution. Model usage is a cost, not a business outcome.

Map the cost of error before setting an accuracy target. A weak marketing draft that a specialist edits is recoverable. A wrong refund, medical instruction, permission decision, or deleted record can be consequential. The NIST AI Risk Management Framework Core explicitly asks teams to document the intended context, expected benefits and costs, knowledge limits, human oversight, and an initial go/no-go decision before deployment.

Turn the claim into a representative evaluation set

Build cases before building the interface. Start with real workflow data that you are permitted to use, then remove or transform sensitive content where possible. Sample across user types, languages, input length, data quality, and business importance. Include ordinary work, rare but costly cases, missing context, conflicting evidence, adversarial instructions, and situations where the correct behavior is to abstain or escalate.

Keep a development set for iteration and a held-out set for deciding whether the change is actually better. Google's responsible AI evaluation guidance recommends product-specific safety data in addition to broad benchmarks, with diverse explicit and implicit adversarial cases. Public benchmarks can compare general capabilities; they cannot tell you whether a feature follows your permissions, policy, terminology, or failure process.

Write the scoring rule before seeing candidate outputs. Exact comparison works for normalized extraction. Classification needs class-specific error review, not only overall accuracy. Open-ended answers need a rubric with observable criteria such as correctness, evidence support, completeness, and appropriate abstention. Use domain reviewers for claims that require domain judgment, and measure reviewer agreement. If capable reviewers cannot agree on a good answer, the task is not ready for automated grading.

Test the smallest credible system against the baseline

Run a deterministic baseline and a strong prompted-model baseline on the same cases. Add retrieval, tools, or multi-step orchestration only when the failure analysis identifies a missing capability. OpenAI's evaluation guidance recommends defining the desired behavior, using representative test inputs, and comparing changes continuously. The general principle is provider-independent: a shared dataset turns model and prompt selection into an engineering comparison rather than a demo contest.

Inspect failures case by case. Did the task lack context? Did retrieval miss the right document? Did the model ignore evidence? Did a parser reject valid output? Did the reviewer change the standard halfway through? These require different fixes. Repeating prompts until one looks good leaks knowledge of the test into the design and produces an optimistic estimate.

Test repeated runs when variability matters. Measure the complete path, including search, reranking, tool calls, validation, and retries—not only the model response. Record the model snapshot, prompt version, retrieval configuration, dataset version, and grader version so a result can be reproduced. Keep evaluation cases in the release process because provider models and application context both change.

Design permissions and data boundaries before integration

List every data class that can enter a prompt or training set: user input, uploaded files, customer records, retrieved passages, logs, and reviewer feedback. For each, document authorization, purpose, retention, region, provider use, deletion, and whether it can appear in telemetry. Verify current vendor terms and configuration rather than assuming that an API or enterprise label provides the boundary you need.

Authorization belongs outside the model. Filter retrieval using the authenticated user and tenant before content reaches the prompt. Keep tool credentials scoped to the user and task. Treat retrieved documents, web pages, emails, tool output, and model output as untrusted data. They can contain instructions designed to redirect the model or exploit downstream software.

OWASP describes excessive agency as the combination of unnecessary tool functionality, excessive permissions, or excessive autonomy. Reduce all three: expose narrow operations, use least-privilege credentials, validate arguments and results, set rate limits, and require human confirmation for consequential actions. A prompt that says “ignore malicious instructions” is not an authorization control.

Specify failure behavior as part of the feature

Define what happens on timeout, rate limiting, malformed output, weak evidence, policy refusal, tool failure, and provider outage. Put a hard deadline around each external call. Retry only transient operations, cap attempts, and make mutations idempotent. Validate structured output at runtime before it reaches business logic. Never interpret confident prose as proof that an action succeeded.

Human review is valuable only when the reviewer has enough context, time, and authority to catch the relevant failure. Show sources and uncertainty, highlight proposed changes, and make rejection or correction easy. Measure whether reviewers actually detect planted errors; a ceremonial approval button can add latency without reducing risk.

Provide a usable fallback. A support agent might continue with search and templates. An extraction job might queue for manual review. A drafting tool might preserve the user's input and explain that generation is temporarily unavailable. If failure destroys work or traps the user, the application is not reliable even when the model usually performs well.

Budget latency and cost per successful task

Set an end-to-end latency budget based on the workflow, then allocate it across retrieval, model calls, tools, validation, and retries. Measure median and tail latency under realistic concurrency. A workflow that feels responsive in a single local test may fail when a slow dependency sits in every request or several agent steps run serially.

Calculate cost per completed useful task, including input and output tokens, embeddings, reranking, tool APIs, retries, storage, observability, evaluation, and human review. Model price per token is not the unit the product sells. Track cost by feature and tenant with limits that prevent a prompt or loop from creating unbounded spend.

Run a small shadow or assisted pilot after offline evaluation. Compare the feature with the baseline, collect corrections and user outcomes, and keep a kill switch. Production feedback should expand the evaluation set, not replace it. Monitor input drift, acceptance, overrides, escalations, latency, cost, and policy events after launch.

Use a written go, narrow, or stop decision

Proceed only when the evidence supports both capability and containment. The technique-selection follow-up is RAG vs fine-tuning vs prompt engineering. The AI workflow shows how evaluation fits into delivery, and the services overview explains the broader product engineering scope.