Capstone: Production-Readiness Review

Where you are in the journey

You have carried one Contoso Travel Agent through the full AgentOps operating model. In Lab 1 you built travel-agent:1 in Microsoft Foundry and created the agentops-vbd/ workspace. In Lab 2 you evaluated it. In Lab 3 you proved a regression gate with travel-agent:2. In Lab 4 you made the agent observable. In Lab 5 you added safety and governance evidence. In Lab 6 you closed the improvement loop by promoting production signals back into the evaluation dataset.

The capstone wires those artifacts into CI/CD. You will create a GitHub Actions PR gate that runs evaluation plus Doctor, prove one green PR and one red regression PR, generate the final evidence pack, review it in Cockpit, and sign a ship / no-ship decision.

Duration

90 minutes

You will build

A GitHub Actions workflow generated by the accelerator that runs agentops eval run --baseline and agentops doctor --evidence-pack as a PR gate, plus a final production-readiness evidence pack and signed ship-decision record.

Prerequisites

  • Completed Labs 1-6.
  • Your agentops-vbd/ folder still has:
    • agentops.yaml
    • .agentops/data/travel-smoke.jsonl
    • .agentops/baseline/results.json
    • .agentops/release/latest/evidence.md from earlier labs, if already generated
    • governance files from Lab 5, such as .assert/, acs.yaml, or .agentops/governance/ if your lab environment produced them
  • The local commands below run from the agentops-vbd/ folder.
  • GitHub CLI is installed and signed in, or you know how to create the repository and PRs in the GitHub portal.
  • Azure CLI is installed and signed in locally.
  • Azure credentials are available to GitHub Actions. The recommended production pattern is GitHub Actions OpenID Connect (OIDC) with a Microsoft Entra federated credential. A service principal secret can work for a training tenant, but OIDC is preferred because no long-lived secret is stored in GitHub. Microsoft Learn: Configure an app to trust an external identity provider.
  • The following local environment variables are set before you test locally:
$env:AZURE_AI_FOUNDRY_PROJECT_ENDPOINT = "https://<resource>.services.ai.azure.com/api/projects/<project>"
$env:AZURE_OPENAI_ENDPOINT = "https://<openai-resource>.openai.azure.com"
$env:AZURE_OPENAI_DEPLOYMENT = "gpt-4o-mini"

Concepts in two minutes

  • AgentOps is the operating model. The loop is Evaluate -> Ship -> Observe -> Operate. The accelerator gives you repo-side commands that make the loop repeatable.
  • Foundry is the control plane. Foundry owns the agent, evaluation execution, runtime traces, safety, and drilldown.
  • GitHub Actions is the release gate. A PR should not merge just because code changed. It should merge only when the candidate agent has current evaluation evidence and readiness evidence.
  • Doctor is the release reviewer. agentops doctor --evidence-pack reads evaluation history, telemetry, Foundry configuration, workflow hygiene, and governance artifacts, then writes .agentops/release/latest/evidence.json and .agentops/release/latest/evidence.md.
  • Cockpit is the local command center. agentops cockpit opens a localhost view that brings together eval history, Doctor findings, workflow status, telemetry links, and next actions.
  • Exit codes matter. 0 means pass, 2 means thresholds or configured findings failed, and 1 means a runtime or configuration error.

Step by step

1. Put the workspace under git and push to GitHub

From the folder that contains agentops.yaml, initialize git if needed, create a .gitignore that keeps generated run output out of source control, and commit the release contract files.

cd agentops-vbd

if (-not (Test-Path .git)) {
  git init -b main
}

@"
# AgentOps generated run output
.agentops/results/
.agentops/release/
.agentops/cache/
.agentops/agent/report.md

# Local Python and editor state
.venv/
__pycache__/
.vscode/
"@ | Set-Content .gitignore

$pathsToCommit = @(
  ".gitignore",
  "agentops.yaml",
  ".agentops\data\travel-smoke.jsonl",
  ".agentops\baseline\results.json",
  ".agentops\governance",
  ".assert",
  "acs.yaml"
)

$pathsToCommit |
  Where-Object { Test-Path $_ } |
  ForEach-Object { git add $_ }

git status --short
git commit -m "Add AgentOps release workspace"

If you do not have a remote yet, create a private GitHub repository and push:

gh repo create agentops-vbd --private --source . --remote origin --push

If the repository already has origin, push the current branch:

git push -u origin main

Expected result: GitHub has a repository containing agentops.yaml, the smoke dataset, the baseline results, and governance artifacts. Generated folders such as .agentops/results/ and .agentops/release/ are not committed.

2. Analyze CI readiness and generate the PR gate workflow

Run the workflow analyzer first. The accelerator docs describe this as a read-only check that inspects repository shape, azure.yaml, Bicep files, agentops.yaml, Foundry prompt-agent shape, source-controlled prompt files, landing-zone signals, private-network terms, Docker or Container Apps signals, and existing CI folders. It also recommends the evaluation runner.

agentops workflow analyze

Generate the PR workflow. The GitHub Actions guide recommends generating the PR gate first, then adding deployment stages after environments and Azure OIDC are ready.

agentops workflow generate --kinds pr --doctor-gate critical --force

Open the generated workflow:

code .github\workflows\agentops-pr.yml

Walk the file with the group. The generated GitHub template is grounded in the public docs/ci-github-actions.md guide and includes the same building blocks:

name: AgentOps PR

on:
  pull_request:
    branches:
      - develop
      - "release/**"
      - main

permissions:
  contents: read
  pull-requests: write
  id-token: write

jobs:
  eval:
    name: AgentOps eval (PR gate)
    runs-on: ubuntu-latest
    environment: dev
    steps:
      - name: Checkout
        uses: actions/checkout@v6

      - name: Azure login (OIDC)
        uses: azure/login@v3

      - name: Install AgentOps accelerator
        run: |
          uv pip install --system "agentops-accelerator[foundry,agent]"

      - name: Run AgentOps eval
        id: eval
        run: |
          set +e
          BASELINE_ARG=""
          if [ -f .agentops/baseline/results.json ]; then
            BASELINE_ARG="--baseline .agentops/baseline/results.json"
          fi
          agentops eval run --config "agentops.yaml" $BASELINE_ARG
          ec=$?
          echo "exit_code=$ec" >> "$GITHUB_OUTPUT"
          if [ $ec -eq 0 ]; then
            echo "result=pass" >> "$GITHUB_OUTPUT"
          elif [ $ec -eq 2 ]; then
            echo "result=threshold_failed" >> "$GITHUB_OUTPUT"
          else
            echo "result=error" >> "$GITHUB_OUTPUT"
          fi
          exit $ec

      - name: Generate release evidence
        if: always()
        run: |
          agentops doctor --workspace . --out .agentops/agent/report.md --severity-fail critical --evidence-pack

      - name: Upload AgentOps results
        if: always()
        uses: actions/upload-artifact@v7
        with:
          name: agentops-pr-results

Commit and push the workflow:

git add .github\workflows\agentops-pr.yml
git commit -m "Add AgentOps PR gate workflow"
git push

Expected result: .github/workflows/agentops-pr.yml exists in the repository. It installs the accelerator, signs in to Azure, runs evaluation, runs Doctor with --evidence-pack, uploads agentops-pr-results, writes the run summary, and comments the report on the PR.

3. Configure CI auth and repository settings

Configure the generated workflow to authenticate to Azure. Prefer OIDC. The generated workflow requests id-token: write and uses azure/login, which expects Azure identity values from GitHub Actions variables.

```portal+GitHub clicks GitHub repository > Settings > Secrets and variables > Actions > Variables > New repository variable

Add:

  • AZURE_CLIENT_ID
  • AZURE_TENANT_ID
  • AZURE_SUBSCRIPTION_ID
  • AZURE_AI_FOUNDRY_PROJECT_ENDPOINT
  • AZURE_OPENAI_ENDPOINT
  • AZURE_OPENAI_DEPLOYMENT

Optional, if App Insights cannot be auto-discovered:

  • APPLICATIONINSIGHTS_CONNECTION_STRING ```

Create the environment used by the generated PR workflow:

```portal+GitHub clicks GitHub repository > Settings > Environments > New environment Name: dev Save protection rules appropriate for your training environment.


Create the Microsoft Entra federated credential for GitHub Actions:

```portal+GitHub clicks
Azure portal > Microsoft Entra ID > App registrations > your CI app > Certificates and secrets > Federated credentials > Add credential
Scenario: GitHub Actions deploying Azure resources
Organization: your GitHub organization or username
Repository: agentops-vbd
Entity type: Environment
Environment name: dev
Save

Assign the CI identity the roles needed by the Foundry prompt-agent gate:

```portal+GitHub clicks Azure portal > Foundry project or Azure AI Services resource > Access control (IAM) > Add role assignment Add Foundry User to the CI identity.

Azure portal > Azure AI Services account that hosts the evaluator model > Access control (IAM) > Add role assignment Add Cognitive Services OpenAI User to the CI identity.


If your organization uses a service principal secret instead of OIDC for this training lab, store the secret in GitHub Actions Secrets and update the workflow according to the `azure/login` action guidance. Do not commit secrets to the repository.

Expected result: GitHub Actions can request an Azure token, read the Foundry project endpoint, call the evaluator model deployment, and write AgentOps evidence artifacts during a PR run.

### 4. Prove the gate on a real PR - green path

Create a harmless change that does not alter agent behavior. A YAML comment is enough for this demonstration.

```powershell
git switch main
git pull
git switch -c feature/capstone-green

Add-Content agentops.yaml "`n# Capstone green PR demonstration"

git add agentops.yaml
git commit -m "Prove AgentOps PR gate stays green"
git push -u origin feature/capstone-green

gh pr create --base main --head feature/capstone-green --title "Prove AgentOps PR gate stays green" --body "Green capstone PR proving eval plus Doctor gate on a harmless change."

Watch the run:

```portal+GitHub clicks GitHub repository > Pull requests > Prove AgentOps PR gate stays green > Checks Open AgentOps PR / AgentOps eval (PR gate) Open Summary and Artifacts Download agentops-pr-results if you want to inspect results.json, report.md, evidence.json, and evidence.md.


Expected result: The PR check passes. The run summary shows the AgentOps PR Eval report and evidence summary. The run has an `agentops-pr-results` artifact.

### 5. Prove the gate catches a regression - red path, then fix it

Now demonstrate the headline lesson of the workshop: a changed agent version can be blocked before merge. Use the Lab 3 regression candidate, `travel-agent:2`.

```powershell
git switch main
git pull
git switch -c feature/capstone-red-regression

(Get-Content agentops.yaml) -replace 'agent: "travel-agent:1"', 'agent: "travel-agent:2"' | Set-Content agentops.yaml

git add agentops.yaml
git commit -m "Demonstrate regressed travel agent gate failure"
git push -u origin feature/capstone-red-regression

gh pr create --base main --head feature/capstone-red-regression --title "Demonstrate AgentOps regression gate" --body "Red capstone PR proving the baseline comparison blocks a regressed agent."

Open the failed check:

```portal+GitHub clicks GitHub repository > Pull requests > Demonstrate AgentOps regression gate > Checks Open AgentOps PR / AgentOps eval (PR gate) Read the failed evaluation step and the run summary. Open Artifacts > agentops-pr-results. Open report.md and evidence.md.


The evaluation command uses the AgentOps exit-code contract. A threshold or baseline failure exits `2`, so the PR check fails and merge is blocked when branch protection requires the AgentOps PR check.

Fix the regression by returning to the passing agent version:

```powershell
(Get-Content agentops.yaml) -replace 'agent: "travel-agent:2"', 'agent: "travel-agent:1"' | Set-Content agentops.yaml

git add agentops.yaml
git commit -m "Restore passing travel agent version"
git push

Expected result: The first run on this PR fails with exit code 2, proving the release gate catches the regression. After the fix commit, the PR check reruns and passes.

6. Generate the final production-readiness evidence

Return to the passing main branch and generate the final evidence pack locally. Doctor writes the release evidence under .agentops/release/latest/.

git switch main
git pull

agentops eval run --baseline .agentops\baseline\results.json
agentops doctor --evidence-pack

code .agentops\release\latest\evidence.md

Use this mapping during the review:

Evidence section Pillar What the reviewer is checking
Latest eval result and threshold summary Evaluate The candidate was tested against the committed dataset and thresholds.
Baseline comparison and regression findings Evaluate / Ship The candidate did not regress against .agentops/baseline/results.json.
Workflow and PR gate evidence Ship CI/CD now runs evaluation plus Doctor before merge.
Telemetry and monitoring readiness Observe Lab 4 telemetry is connected or the evidence calls out the gap.
Safety, governance, and responsible AI findings Operate Lab 5 artifacts and safety signals are present for release review.
Trace-regression and improvement-loop signals Operate Lab 6 production learning is part of the dataset and future gates.

Expected result: .agentops/release/latest/evidence.json and .agentops/release/latest/evidence.md exist. The Markdown file gives you a reviewer-friendly production-readiness summary.

7. Run the operator review in Cockpit

Open Cockpit from the passing workspace.

agentops cockpit

Walk the local command center in this order:

```portal+GitHub clicks Cockpit > Foundry connection Confirm the project endpoint and agent target are present.

Cockpit > Eval gate summary Confirm the latest evaluation is green and the baseline comparison is visible.

Cockpit > AgentOps Doctor Confirm there are no blocking critical findings for the ship decision, or record the exception.

Cockpit > Observability readiness Confirm telemetry imported in Lab 4 is visible or linked for drilldown.

Cockpit > CI/CD Pipelines Confirm the GitHub Actions PR gate exists and recent runs are visible.

Cockpit > Next actions Copy any remaining action into the ship-decision record.


Expected result: Cockpit shows the final operational state: eval green, evidence present, telemetry imported or explicitly reviewed, safety and governance evidence present, and the CI/CD gate visible.

### 8. Fill the ship / no-ship decision record

Copy this template into your workshop notes or into a GitHub PR comment. The decision should cite evidence, not opinions.

```text
# Ship decision record - Contoso Travel Agent

Date:
Reviewer:
Agent candidate:
Repository:
GitHub Actions run:
Evidence pack path: .agentops/release/latest/evidence.md
Cockpit review completed: Yes / No

## Decision
Go / No-go:

## Evidence reviewed
- Evaluation thresholds met:
- Baseline regression gate proven:
- Red regression PR demonstrated and blocked:
- Observability live or explicitly accepted:
- Safety and governance artifacts reviewed:
- Improvement loop demonstrated with promoted traces or regression rows:
- Doctor critical findings:

## Release conditions or follow-up
- Required before ship:
- Required after ship:
- Owner and due date:

## Signed
Name:
Decision timestamp:

For the guided workshop, the expected decision is Go only if all of these are true:

  • agentops eval run --baseline .agentops\baseline\results.json passes.
  • A green PR proved the gate works for a harmless change.
  • A red PR proved the gate blocks travel-agent:2 or an equivalent regression with exit code 2.
  • agentops doctor --evidence-pack produced final evidence.
  • Cockpit review found no unaccepted critical issue.
  • Observability, safety, governance, and improvement-loop artifacts from Labs 4-6 are either present or explicitly documented as accepted gaps.

Expected result: You have a completed ship-decision record that states Go or No-go, cites the evidence pack, cites the green and red CI runs, and names any remaining follow-up owners.

Checkpoint

Run this final local verification:

Test-Path .github\workflows\agentops-pr.yml
Test-Path .agentops\baseline\results.json
Test-Path .agentops\release\latest\evidence.md
git status --short

Expected result:

  • GitHub Actions has at least one green AgentOps PR run.
  • GitHub Actions has at least one red AgentOps PR run that failed because the regression gate returned exit code 2.
  • .agentops/release/latest/evidence.md exists locally from the passing main branch.
  • The attendee has a completed ship / no-ship decision record.

Troubleshooting

Symptom Likely cause Fix
agentops workflow generate did not create the file you expected The command generated a different kind set or wrote to a different repository root Run agentops workflow analyze, then run agentops workflow generate --kinds pr --doctor-gate critical --force from the folder that contains agentops.yaml. The PR workflow path should be .github/workflows/agentops-pr.yml.
GitHub Actions fails at Azure login (OIDC) Missing GitHub environment, wrong federated credential subject, or incorrect AZURE_CLIENT_ID, AZURE_TENANT_ID, or AZURE_SUBSCRIPTION_ID Confirm the workflow uses environment: dev. Create the dev GitHub environment. In Microsoft Entra, create a federated credential for the same repository and environment. Check the repository variables.
Evaluation in CI returns no usable metrics or authorization errors The CI identity can sign in but lacks Foundry or evaluator-model permissions Assign Foundry User on the Foundry project or resource and Cognitive Services OpenAI User on the Azure AI Services account that hosts AZURE_OPENAI_DEPLOYMENT.
CI cannot find the baseline .agentops/baseline/results.json was not committed, or the workflow was edited to look at a different path Commit .agentops/baseline/results.json. Keep the path aligned with agentops eval run --baseline .agentops/baseline/results.json.
Evidence artifact is missing Doctor did not run, failed before writing evidence, or upload-artifact path was edited Confirm the workflow has a step that runs agentops doctor --workspace . --out .agentops/agent/report.md --severity-fail critical --evidence-pack with if: always(). Confirm upload-artifact includes .agentops/release/latest/evidence.json and .agentops/release/latest/evidence.md.
The red PR fails but GitHub still allows merge Branch protection is not requiring the AgentOps PR check GitHub repository > Settings > Branches > Add branch protection rule for main. Enable Require status checks to pass and select AgentOps PR / AgentOps eval (PR gate).
The harmless green PR fails with exit code 1 Runtime or configuration error, not a quality failure Read the failing step logs first. Check environment variables, OIDC login, Foundry endpoint, dataset path, and model deployment name.
The regression PR does not fail agentops.yaml did not actually point to travel-agent:2, the Lab 3 regression is not deployed, or thresholds are too loose Open agentops.yaml in the PR diff. Confirm agent: "travel-agent:2". Re-run agentops eval run --baseline .agentops\baseline\results.json locally and inspect .agentops/results/latest/report.md.

What you just learned

  • How the full AgentOps loop becomes automated CI/CD: Evaluate -> Ship -> Observe -> Operate.
  • How to use a baseline comparison and exit code 2 as a real merge gate.
  • How Doctor evidence turns many readiness checks into a release-review artifact.
  • How Cockpit acts as the local command center for operator sign-off.
  • How to make a ship decision from evidence instead of intuition.

Carry forward

Your final artifact is the signed ship / no-ship decision. Apply the same pattern to your own Foundry agent: expand the dataset from production traces, keep the baseline current, require the PR gate, review Cockpit before release, and operationalize the cadence for continuous improvement.

This site uses Just the Docs, a documentation theme for Jekyll.