Would You Approve This AI Release? A Worked Case on Changes After Evaluation

In short: an AI evaluation supports only the version it tested. When a team changes the system after the evaluation passes, compare the tested manifest with the release candidate, sort each piece of evidence into valid, stale, missing, failed or uncertain, retest what the changes touch, and have one named owner approve, hold or escalate. This case takes a fictional release through every step.

Updated November 8, 2026. We checked every source on this page that day.

Who this is for. Readers in audit, privacy, risk or compliance who want to show AI assurance judgment, and hiring managers who need a fair work sample. Start with a skill you already use. The AI Assurance Engineering Skills Map shows how to turn it into a portfolio project you can explain in an interview, and this case is a worked version of its Project 2, the retesting rule, and Project 3, the release gate.

On this page:

• A worked release case that names each version and each owner.
• A completed one-page release decision.
• A blank decision form as a printable PDF and a spreadsheet.
• The interview question, with what a strong answer explains.

What Happened Between the Evaluation and the Release Request?

The team evaluated release H-3 on Day 1 and it passed. Over the next two weeks five things changed, four of which can alter what the assistant says or does. On Day 16 the team asked to ship the result as H-3.1. Everything in this case is fictional: Harbor Services, its system, the versions, the days and the results.

Harbor Services is the fictional company from the Skills Map. Its internal document assistant drafts summaries for staff from approved documents and can create draft tickets. It cannot approve payments or change customer records. Release H-2 is in production. We assume no legal risk classification for the assistant; the EU AI Act section explains what would change if it were a high-risk AI system.

Harbor timeline (fictional):

• Day 1: evaluation EV-H3 runs on candidate H-3 and passes every mandatory check.
• Day 3: the product team rewrites the prompt, P-5 to P-6, so each summary ends with a list of action items. P-6 also adds two worked examples.
• Day 6: the data team adds the HR policy library to the retrieval index, R-7 to R-8. Its 1,400 documents carry a new access label, HR-restricted.
• Day 9: the model provider moves the alias assist-large to a newer model snapshot. Harbor's configuration points at the alias, so the next deploy picks up the new snapshot.
• Day 12: operations asks for a new tool permission, T-3 to T-4. The assistant could set ticket priority and route tickets to a team queue with no staff approval.
• Day 14: the staff portal button changes from Summarize to Draft summary.
• Day 16: the team asks to release H-3.1, citing "evaluation passed on Day 1."

Which Version Did the Evaluation Actually Cover?

The evaluation covered the components recorded in its manifest on the day it ran, not whatever now carries the release name. Read that manifest line by line against the candidate. If a component is recorded only by a moving name, such as a model alias, the record cannot prove which version was tested.

Harbor example (fictional). The EV-H3 record lists:

• Model: provider alias assist-large, with no dated snapshot recorded.
• Retrieval index: R-7.
• Prompt: P-5.
• Tool policy: T-3, which allows draft tickets only. Staff approve every ticket.
• Governance policy: G-5.
• Tests: test set TS-3 (300 staff documents), 40 leakage probes, and the ticket access-boundary suite.

The report's title declares that H-3 passed. The manifest shows that H-3 meant R-7, P-5 and T-3. Harbor can infer from the provider's change notice that the Day 1 run used the earlier snapshot, but its own record does not prove it, and H-3.1 would run on the newer one.

Why the manifest matters. A research preprint on reusing assurance evidence states the condition plainly: reuse is safe only if each control's declared dependencies cover everything that can change its outcome, and only if the release gate checks that the reused evidence describes the candidate being released (Mukhopadhyay and Nalla, arXiv 2610.27672, version 1, abstract). In the authors' replay of 38,083 commits from 714 public Data Version Control pipelines, 3.3% of commits reused a stage whose imported code or referenced configuration had changed, and evaluation stages were the most exposed (same source). Their sample is public repositories, so the 3.3% describes that sample and says nothing direct about failure rates in company AI systems.

What to ask for:

• The evaluation manifest: every component with a version or hash, the test set version, the run date and who ran it.
• The release candidate's manifest in the same format.
• A diff of the two, produced by tooling where possible rather than from memory.
• For any vendor model, the dated snapshot the provider published. An alias can point at a different model tomorrow.

The EU AI Act asks for the same discipline in the technical documentation of high-risk systems: "the version of the system reflecting its relation to previous versions" (Regulation (EU) 2024/1689, Annex IV, point 1(a), consolidated text of July 27, 2026).

What Changed After the Evaluation Passed?

Five things changed between Day 1 and Day 16. Four can change what the assistant says or does, and one cannot. Classify each change by what it can affect, not by how large it looks in the change ticket.

Two sources explain why the prompt change cannot be waved through. NIST's Generative AI Profile says gaps between benchmarks and real-world use of generative AI "may likely be exacerbated due to prompt sensitivity" (NIST AI 600-1, Appendix A.1.4, page 49). And in a November 2026 preprint, eight configurations of one model scored between 68.2% and 70.3% accuracy while 24.3% of questions received different labels across them (Yorke, arXiv 2611.06655, version 1, abstract). Our same accuracy, different decisions walkthrough shows how that plays out in a release decision.

Which Evidence Still Holds?

A piece of evidence still holds only when everything it depends on is unchanged, or when a written equivalence note shows the change cannot affect it. In Harbor's case two of six records still hold for H-3.1, three are stale or missing, and one is missing because a requested change removes the control it tested.

The 30-minute target and the 90-day window are Harbor's own fictional choices. No source cited here sets them.

How to set each status:

• Valid: the record names the exact components the candidate uses, an authorized person produced it, the result passed, and it is inside its validity period.
• Stale: something the test depends on has changed since it ran.
• Missing: no record covers this control for this candidate.
• Failed: the test ran on this candidate and did not pass.
• Uncertain: you cannot tell what the test depended on. Keep it visible rather than rounding it up to valid.

What Needs Retesting Before Release?

Retest each mandatory check whose evidence is stale or missing, on the exact candidate. Leave the rest alone and write down why. Harbor reruns two tests, adds one review, defers two retests until the committee rules, and writes one equivalence note.

Harbor retest plan (fictional):

• Pin the model to a dated snapshot in the manifest, then rerun E1 on that snapshot with P-6 and R-8. Define an HR policy slice before the run, with its own pass rule.
• Rerun E2 with new probes: requests for HR-restricted documents from accounts outside HR.
• The privacy lead reviews the HR policy library before R-8 goes live, which closes E5.
• Defer the E3 and E4 retests until the committee rules on T-4. If T-4 is dropped, E3 and E4 stand as they are for T-3.
• No retest for the button label. Record the equivalence note: the label is not sent to the model.
• Update the rollback manifest to include R-8 and P-6, then keep E6.

The rule behind this plan comes from the Skills Map: when a candidate changes an input used by a mandatory control, the earlier result cannot be reused unless a documented equivalence rule shows the change cannot affect that control (Project 2). NIST's AI RMF asks that performance be "demonstrated for conditions similar to deployment setting(s)" and documented (AI RMF 1.0, MEASURE 2.3, page 29). Adding the HR library also fits the Generative AI Profile's action to "re-evaluate risks when adapting GAI models to new domains" (NIST AI 600-1, MP-4.1-008, page 26).

Should You Approve, Hold or Escalate?

Hold H-3.1, escalate the tool permission change, and approve a smaller release once the retests pass. Hold when the gaps are ones the release owner can close; escalate when a change needs a decision the release owner is not allowed to make.

These rules are GRC Careers editorial guidance. No standard sets them.

Harbor decision (fictional):

• Hold H-3.1 as requested.
• Escalate T-4 to the AI Governance Committee. Policy G-5 requires staff approval for every ticket action, and the release owner cannot waive it.
• Approve H-3.2, which is H-3.1 without T-4, once E1, E2 and E5 pass and the rollback manifest is updated.

Splitting the release this way ships what the evidence covers and sends the rest to the people who can decide it. NIST's AI RMF describes the same moment as a determination of "whether its development or deployment should proceed" (AI RMF 1.0, MANAGE 1.1, page 32), and lists the response options as mitigating, transferring, avoiding or accepting (MANAGE 1.3, same page).

Who Owns the Release Decision?

One accountable role decides, and the record names it before anyone signs. In Harbor's case the Head of Service Operations owns the release. The people who produce or review evidence advise, and the committee decides only what was escalated to it.

Harbor roles (fictional, roles not people):

• Release owner, accountable: Head of Service Operations, who signs the decision.
• Evidence producer: Technical Assurance Lead. Runs E1 and E2 and cannot approve the release.
• Privacy sign-off: Privacy Lead, for E5.
• Escalation: AI Governance Committee, chaired by the Chief Risk Officer. Rules on T-4 and on any change to policy G-5.
• Independent check: internal audit samples the decision record afterward and does not sign the release.

The last point follows the Skills Map's independence note: internal audit may advise on control design, but operational management owns the control and accepts the business risk (Role Map 2). NIST's AI RMF asks that roles, responsibilities and lines of communication for AI risk be documented and clear, and that executive leadership take responsibility for decisions about risks of AI development and deployment (AI RMF 1.0, GOVERN 2.1 and GOVERN 2.3, page 23). It also calls for internal experts who did not serve as front-line developers, or independent assessors, in regular assessments (MEASURE 1.3, page 29).

What Does a Completed One-Page Release Decision Look Like?

It names the evaluated manifest, every change since, the status of each piece of evidence, the decision, its conditions and its owner, and the events that would reopen it. This is the record Harbor's release owner signed on Day 16. Everything in it is fictional.

Download this completed example (PDF, one page)

Is This a Substantial Modification Under the EU AI Act?

Only a high-risk AI system can trigger the Act's substantial-modification rule, and only through a change made after it was placed on the market or put into service. Harbor's assistant is not classified, so the rule does not apply in this case. If the assistant were a high-risk AI system already in service, a substantial modification would require a new conformity assessment.

What the text says, from the consolidated text of July 27, 2026, which includes the amendments made by Regulation (EU) 2026/1744:

• Definition: a change after placing on the market or putting into service "which is not foreseen or planned in the initial conformity assessment" and that affects compliance with the Chapter III, Section 2 requirements or changes the intended purpose for which the system was assessed (Article 3, point 23).
• Consequence: high-risk systems already assessed "shall undergo a new conformity assessment procedure in the event of a substantial modification," whether or not the system is distributed further (Article 43(4), first subparagraph).
• Planned changes: for high-risk systems that continue to learn, changes pre-determined at the initial conformity assessment and described in the technical documentation are not a substantial modification (Article 43(4), second subparagraph, with Annex IV, point 2(f)).
• Who becomes the provider: a distributor, importer, deployer or other third party that makes a substantial modification to a high-risk system already on the market or in service, so that it stays high-risk, is treated as its provider (Article 25(1)(b)).
• Records: the technical documentation includes test logs and test reports "dated and signed by the responsible persons" (Annex IV, point 2(g)) and a description of relevant changes made through the lifecycle (Annex IV, point 6).
• Logs: logging must allow events relevant to identifying situations that may result in a substantial modification to be recorded (Article 12(2)(a)).
• Timing: the high-risk requirements in Chapter III, Sections 1 to 3 apply from December 2, 2027 for systems classified under Annex III and from August 2, 2028 for systems classified under Annex I (Article 113, point (c)).
• Guidance to watch: the Commission must develop guidelines on the practical implementation of the substantial-modification provisions (Article 96(1)(c)). Check whether they have been published before you rely on any reading of the definition.

Our reading, not legal advice. The internal question (does our evidence cover this candidate?) and the legal question (is this change a substantial modification?) are separate. A prompt change can make test evidence stale without being a substantial modification, and Harbor would still hold the release. Whether a specific change is substantial is a legal determination for counsel, made on the facts and on the text in force at the time.

What Do NIST's AI Frameworks Say About Changes After Evaluation?

NIST's AI Risk Management Framework is voluntary and sets no retest rule, but several of its subcategories line up with this decision, and its Generative AI Profile calls for re-evaluation when models are enhanced or moved into new domains. NIST says AI RMF 1.0 is being revised, so check for a newer edition before you cite it.

• MANAGE 1.1: a determination of whether the system achieves its intended purposes and whether development or deployment should proceed (AI RMF 1.0, Table 4, page 32).
• MEASURE 2.1 and 2.3: test sets, metrics and tools are documented, and performance is demonstrated for conditions similar to deployment (Table 3, page 29).
• MANAGE 4.1: post-deployment monitoring plans include change management (Table 4, page 33).
• MP-4.1-007: "Re-evaluate models that were fine-tuned or enhanced on top of third-party models" (NIST AI 600-1, page 26).
• MP-4.1-008: re-evaluate risks when adapting generative AI models to new domains, and set up warnings for when earlier assumptions may no longer hold (same page).
• MS-2.8-003: version control systems can track changes across the AI lifecycle (NIST AI 600-1, page 34).

Currency: AI RMF 1.0 was released on January 26, 2023 and the Generative AI Profile on July 26, 2024. On November 8, 2026, NIST's AI RMF page said "The AI RMF 1.0 is being revised as part of the White House AI Action Plan."

We do not map this exercise to ISO/IEC 42001 clauses. For how that management-system standard relates to the AI Act, see our ISO/IEC 42001 and EU AI Act comparison.

Which Skills Does This Case Show for Audit, Privacy, Risk and Compliance Work?

It shows that you can test whether evidence matches what is actually being released, a judgment audit, privacy, risk and compliance work already uses. Start from your own field and point an interviewer to the part of the case that proves it.

The technical side of the same work is covered in the AI evaluation and testing skills guide, and the audit side in the AI auditing skills guide.

Interview Question: The System Passed, Then Changed. What Would You Ask to See?

"An AI system passed its evaluation, but the team changed it afterward. What would you ask to see before approving the release?"

A strong answer explains:

• Which exact version the evaluation covered, read from its manifest, including the dated model snapshot rather than an alias.
• Every change since then, from a diff: prompt, data or retrieval index, model version, tools and permissions, code, and deployment context.
• For each piece of evidence, whether it still holds and why, with written equivalence notes for changes that cannot affect it.
• Which checks to rerun on the exact candidate, including slices defined before the run, rather than a rerun of the headline score alone.
• Who decides, and when the decision must go above the release owner: a policy conflict, a change of purpose or risk tier, a risk acceptance beyond the owner's authority, or owners who disagree.
• What would hold the launch, what would reopen an approval, and which monitoring signal would show trouble after release.

A weak answer says "it passed, so ship it," or asks for the whole evaluation to be rerun without saying what changed or which results the change can affect.

Follow-up an interviewer can add. Change one fact: the provider confirms the snapshot behind assist-large did not change. Ask which statuses move. A strong candidate says the model is no longer a reason to rerun E1, but the prompt and index still are, so E1 stays stale. The Skills Map's take-home approach uses the same move.

Download the Blank Release Decision Form

The blank form has the same fields as Harbor's record. Fill it in on paper or in the spreadsheet, which has drop-down lists for evidence status and the decision.

• Blank release decision form (PDF, one page, US Letter)
• Blank form plus the Harbor example (spreadsheet, XLSX)
• Blank form (CSV, opens in any spreadsheet app)
• Completed Harbor example (PDF, one page)

The form is an editorial work sample from GRC Careers. It is not a standard, and a completed copy is not a compliance record.

How Do You Turn This Case Into a Portfolio Piece?

Change one fact at a time and redo the record. A reviewer should be able to reproduce your decision from the page without asking you anything.

• The provider confirms the snapshot did not change. Which statuses move, and which stay?
• The E2 rerun returns one HR-restricted document to a finance account. What happens to H-3.2, and who is told?
• Write the escalation memo for T-4 in under 300 words: the request, the policy it conflicts with, the options, and the decision the committee needs to make.
• Add an audit step: sample three past releases and check whether each one's evaluation manifest matched what was deployed.

Label everything as a simulation and use synthetic data. Never describe a learning exercise as a production system or as proof of legal compliance. When it is done, add it to the four-week route in the Skills Map, and see open AI assurance jobs for the roles that ask for this work.

Sources and What We Checked

All sources were checked on November 8, 2026.

1. Regulation (EU) 2024/1689 (the AI Act), consolidated text of July 27, 2026, incorporating Regulation (EU) 2026/1744. Articles 3(23), 12(2)(a), 25(1)(b), 43(4), 96(1)(c) and 113(c); Annex IV, points 1(a), 2(f), 2(g) and 6. EUR-Lex consolidated text. Read through the Ansvar gateway, which served this consolidation as current.
2. NIST AI 100-1, Artificial Intelligence Risk Management Framework (AI RMF 1.0), January 2023. GOVERN 2.1 and 2.3 (page 23), MEASURE 1.3, 2.1 and 2.3 (page 29), MANAGE 1.1 and 1.3 (page 32), MANAGE 4.1 (page 33). NIST AI 100-1 (PDF).
3. NIST AI 600-1, Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile, July 2024. MP-4.1-007 and MP-4.1-008 (page 26), MS-2.8-003 (page 34), Appendix A.1.4 (pages 48 to 49). NIST AI 600-1 (PDF).
4. NIST, AI Risk Management Framework page, for release dates and revision status. nist.gov.
5. Atanu Mukhopadhyay and Shravya Reddy Nalla, "Reusing Assurance Evidence in AI Pipelines: Validity Conditions, a Fail-Closed Release Gate, and a Replay of 714 Public Data Version Control Pipelines," arXiv 2610.27672, version 1, October 20, 2026. Abstract. arXiv, a research preprint.
6. Louise Yorke, "Beyond accuracy: what AI assurance needs to establish in public services," arXiv 2611.06655, version 1, November 4, 2026. Abstract. arXiv. One task and one model family.

Harbor Services, its system, versions, days, results and roles are fictional. The decision rules, form, interview guidance and portfolio tasks are GRC Careers editorial recommendations.

Where to go next

• Browse the jobs that use these skills
• Follow a career roadmap into the role you want
• Hiring for this? Start from a job description template
• Free certification study games, 652 practice questions

Take it with you

Enter your email and we will send you new guides as they publish. Your download starts either way, and we never sell or share your address with anyone.

Email me new guides

Download the PDF

No email needed to download. We ask because the guides keep coming.

Frequently Asked Questions

Does a passed AI evaluation still count after the system changes?

Only for the parts the change cannot affect. Compare the evaluated manifest with the release candidate, and treat any result that depends on a changed component as stale until it is rerun or a written equivalence note shows the change cannot affect it.

Is a prompt change enough to require a retest?

Usually, for every check that depends on model output. NIST's Generative AI Profile notes that prompt sensitivity can widen the gap between benchmarks and real-world use (NIST AI 600-1, Appendix A.1.4). A change that never reaches the model, such as a button label, needs only a written equivalence note.

Who should sign off an AI release?

One named accountable owner, usually the business owner of the system, after the people who produced and reviewed the evidence record their results. A change that conflicts with policy or alters the system's purpose goes to whoever owns that policy, such as an AI governance committee.

More in this series

• 9 Essential Data Governance Skills for the AI Era
• 10 Internal Audit Skills for Modern Assurance Careers
• 12 Transferable GRC Skills You May Already Have
• Technical vs. Nontechnical GRC Skills: What Employers Actually Need
• AI Governance Skills Employers Actually Hire For
• GRC Analyst Skills: What the Job Actually Requires
• Compliance Analyst Skills
• Risk Assessment Skills
• Controls Testing Skills
• Policy Writing Skills
• Regulatory Change Management Skills
• Third-Party Risk Skills
• Model Risk Management Skills
• AI Impact Assessment Skills
• AI Auditing Skills
• AI Evaluation and Testing Skills for Governance Careers
• Data Lineage Skills
• Data Quality Skills
• Privacy Engineering Skills
• AI Security Skills
• AI Incident Response Skills
• Governance Program Management Skills
• Stakeholder Communication Skills
• Executive Risk Reporting Skills
• Evidence Documentation Skills
• Control Mapping Skills
• Framework Crosswalking Skills
• Vendor Due Diligence Skills
• Responsible AI Skills
• GRC Tools and Automation Skills
• How to Build the 9 Data Governance Skills: A 12-Month Career Plan
• 10 In-Demand Compliance Skills for Today’s GRC Careers
• 10 Essential Risk Management Skills Employers Value
• 10 In-Demand Data Privacy Skills for Privacy Professionals
• 10 Cybersecurity Skills for GRC and Security Careers
• 9 AI Assurance Skills for Audit, Risk, and Governance Professionals
• Writing a Resume for AI Cybersecurity Jobs
• AI Assurance Engineering Skills Map: Four Role Families and Five Portfolio Projects
• The First 24 Hours of a Ransomware Attack