How It Works¶

This document is the single source of truth for understanding the AgentOps architecture. Read it before making any changes.

What Is AgentOps?¶

AgentOps is a standalone Python CLI, local Cockpit, and skill set that helps teams answer the Foundry release question: can we ship this agent, and where is the proof? It:

Reads the flat agentops.yaml configuration plus JSONL dataset rows.
Executes evaluation against a target (Foundry agent, model deployment, HTTP endpoint, or local adapter).
Produces normalized outputs: results.json (machine-readable) and report.md (human-readable).
Returns CI-friendly exit codes: 0 pass, 2 threshold failure, 1 error.
Writes release evidence with agentops doctor --evidence-pack.

Foundry owns agent creation, deployment, runtime, traces, monitoring, red-teaming, datasets, and Microsoft-hosted evaluation drilldown. AgentOps references the candidate those tools produced and adds the repo-controlled release proof: config, gates, artifacts, PR reports, Doctor diagnostics, release evidence, trace-to-regression promotion, and Cockpit links back to Foundry/Azure Monitor.

Key Principles¶

Principle	What It Means in Practice
Thin CLI	`cli/app.py` parses args and delegates to services.
Core is pure	`core/` transforms data without Azure imports or network calls.
Lazy Azure imports	Azure SDK imports stay inside runtime functions.
Pydantic v2	YAML configs and JSON outputs use Pydantic models.
pathlib.Path only	No raw string paths anywhere in the codebase.
No global state	No singletons, no module-level side effects.

Source Code Layout (src layout)¶

src/
└── agentops/
    ├── __init__.py            # Package root (version only)
    ├── __main__.py            # Enables `python -m agentops`
    │
    ├── cli/
    │   └── app.py             # Typer CLI definition (init, eval analyze/run/promote-traces,
    │                              # workflow analyze/generate, skills, mcp, agent)
    │
    ├── core/                  # Pure data layer - no Azure imports, no I/O
    │   ├── agentops_config.py # Flat 1.0 `agentops.yaml` Pydantic schema
    │   ├── dataset_source.py  # Local/Blob/ADLS reference validation
    │   ├── config_loader.py   # YAML → AgentOpsConfig
    │   ├── evaluators.py      # Evaluator catalog (presets + auto-selection)
    │   ├── release_evidence.py # ReleaseEvidence schema
    │   └── results.py         # RunResult / RowResult / TargetInfo / RunSummary
    │
    ├── pipeline/              # Run orchestration - ADD execution flows here
    │   ├── orchestrator.py    # End-to-end `eval run` driver
    │   ├── runtime.py         # Evaluator loading and per-row evaluator execution
    │   ├── invocations.py     # Per-row agent / model invocation strategies
    │   ├── thresholds.py      # Threshold pass/fail evaluation
    │   ├── reporter.py        # Markdown report generation
    │   ├── comparison.py      # Baseline delta rendering for `eval run --baseline`
    │   ├── publisher.py       # Classic Foundry publish (OneDP upload of metrics)
    │   └── cloud_runner.py # New Foundry publish (server-side via OpenAI Evals API)
    │
    ├── services/              # Workspace / project tooling
    │   ├── dataset_source.py  # Read-only Azure snapshot materialization
    │   ├── eval_analysis.py   # `agentops eval analyze` read-only triage
    │   ├── workflow_analysis.py # `agentops workflow analyze` CI/CD triage
    │   ├── initializer.py     # `agentops init` workspace scaffolding
    │   ├── skills.py          # Coding agent skill installation
    │   ├── cicd.py            # CI/CD workflow generation
    │   ├── evidence_pack.py   # Release evidence aggregation/writer
    │   ├── preflight.py       # Pre-flight checks (workspace, auth, Foundry, App Insights)
    │   └── trace_promotion.py # Trace export → dataset candidates
    │
    ├── agent/                 # Doctor and Cockpit
    │   └── observe/           # Hosted/local discovery, bounded queries, OBO, and UI
    ├── mcp/                   # `agentops mcp serve` Model Context Protocol server
    │
    ├── utils/                 # Shared helpers (yaml load, logging, colors)
    │
    └── templates/             # Starter files for `agentops init`
        ├── agentops.yaml      # Minimal flat config (the single config file)
        ├── smoke.jsonl        # Sample dataset
        ├── agent.yaml         # Doctor config seed
        ├── skills/            # Coding agent skill templates
        ├── workflows/         # GitHub Actions workflow templates
        └── pipelines/         # Azure DevOps pipeline templates

Where to Add New Code¶

I want to…	Directory / File
Add a field to `agentops.yaml`	`core/agentops_config.py`
Add a new evaluator preset	`core/evaluators.py` (catalog)
Change pre-flight checks	`services/preflight.py`
Add a target kind	`pipeline/invocations.py` + `core/agentops_config.py`
Tweak the report layout	`pipeline/reporter.py`
Add a publish destination	`pipeline/publisher.py` or `pipeline/cloud_runner.py`
Add a CLI command	`cli/app.py` + a service/pipeline helper
Add a starter template	`templates/` + update `pyproject.toml` package-data
Add a coding agent skill	`templates/skills/<name>/SKILL.md` + sync script
Change hosted Cockpit deployment	`services/cockpit_deployment.py` + `templates/cockpit-hosted/`
Change Observe normalization	`agent/observe/` + pure contracts in `core/observe.py`

Hosted Cockpit and Observe flow¶

agentops cockpit remains local and reads workspace history. The explicit agentops cockpit deploy service resolves the current workspace project, validates an existing single-tenant app registration, renders an azd/Bicep and RBAC/federation preview, then deploys only after confirmation.

flowchart LR
    W["Local workspace project"] --> P["Deployment preview"]
    P --> A["Linux App Service + UAMI"]
    A --> EA["Easy Auth tenant/group boundary"]
    A --> S["Versioned Observe scope"]
    S --> D["ARG + Foundry connection discovery"]
    D --> Q["Bounded Azure Monitor queries"]
    Q --> N["Normalized attributed Observe views"]
    EA --> OBO["Per-user OBO"]
    OBO --> C["Explicit protected trace content"]

The hosted runtime is read-only. Aggregate discovery and query submission use a dedicated UAMI with only Reader and Log Analytics Reader. Explicit protected trace content uses the signed-in user's delegated permission through OBO and is excluded from shared caches. /healthz is the only anonymous route and returns liveness only. Wider Observe scope is represented by canonical ARM IDs in one versioned setting, not by parallel subscription/resource-group variables.

Granular token usage in Models¶

The Cockpit Models view keeps the observed input and output totals and also normalizes three granular token classes when source telemetry emits them: Cache read, Cache write, and Reasoning. These values are observed usage, not billing data, and AgentOps never derives a missing class from another total. A class that was not emitted renders as Not reported; an explicitly reported 0 renders as zero.

Reporting can be incomplete within a row or across the rows aggregated for a telemetry source. An affected model row shows Partial class coverage, while the coverage panel marks the source's token_usage dimension as partial and identifies the classes that were reported and those that were not. This keeps a partially reported aggregate from looking complete without treating missing telemetry as zero.

Eligible, unmapped gen_ai.usage.* token attributes remain visible under their source names. Each row retains at most five, selected deterministically by source attribute name in ascending order. If more are present, the row shows Additional classes truncated; this signal distinguishes omitted classes from attributes that source telemetry never reported. Passthrough attributes do not change normalized values or token-usage coverage.

Request Flow (eval run)¶

For an unfamiliar repo, run agentops eval analyze first. It is a read-only triage step: it inspects agentops.yaml, dataset columns, and local project signals, then tells you whether agentops eval run is ready or whether Copilot should use agentops-config, agentops-dataset, or agentops-eval to adapt the setup.

flowchart TD
    A["agentops init"] --> B["agentops eval analyze"]
    B --> C{"Eval setup ready?"}
    C -->|yes| D["agentops eval run"]
    C -->|no / complex| E["Copilot + AgentOps skills"]
    E --> F["agentops-config / agentops-dataset / agentops-eval"]
    F --> B
    D --> G["results.json + report.md"]
    G --> H["agentops workflow analyze"]
    H --> I{"CI/CD shape clear?"}
    I -->|yes| J["workflow generate --deploy-mode auto"]
    I -->|complex accelerator| K["Copilot + agentops-workflow"]
    J --> L["Deterministic CI/CD"]
    K --> L
    L --> M["agentops doctor"]
    M --> N["doctor --evidence-pack"]
    N --> O["agentops cockpit"]

For AI Landing Zone projects, workflow analyze surfaces the official scripts/Invoke-PreflightChecks.ps1 path when present. Generated azd workflows run that preflight before azd provision, and Doctor reports AI Landing Zone deployment readiness under Operational Excellence so teams can see whether preflight, eval config, workflow deployment, and private-network runner planning are wired together.

When you run agentops eval run, the following happens step by step:

flowchart TD
    A["CLI parses args"] --> B["Load agentops.yaml"]
    B --> C["classify_agent"]
    C --> D["Pre-flight checks"]
    D --> E{"execution mode"}
    E -->|local/cloud| DS{"Local path or Azure Storage URL?"}
    DS -->|local| F["Read JSONL dataset"]
    DS -->|Blob / ADLS| RS["Resolve one bounded temporary snapshot"]
    RS --> F
    F --> G["Invoke target per row or submit Foundry cloud eval"]
    G --> H["Run/collect evaluator scores"]
    E -->|azd| AZD["Call azd ai agent eval using eval.yaml"]
    AZD --> H
    H --> I["Evaluate thresholds"]
    I --> J["Write results.json + report.md"]
    J --> K["Sync .agentops/results/latest"]
    K --> L{"Publish configured?"}
    L -->|Classic| M["Upload local metrics to Foundry"]
    L -->|Cloud| N["Submit server-side Foundry eval"]
    L -->|No| O["Return exit code"]
    M --> O
    N --> O

Exit codes are 0 when all thresholds pass, 2 when one or more thresholds fail, and 1 for runtime or configuration errors.

For AgentOps-owned local, cloud, official-evaluation, and readiness-analysis paths, dataset may identify a local JSONL file or one canonical Blob/ADLS Gen2 HTTPS object. The pure core parser validates the untouched scalar before pathlib.Path coercion, which preserves URLs on Windows. The service layer then uses the process's existing Azure identity to stream one object of at most 100 MiB into a private temporary file. Existing readers consume that stable snapshot, while results, reports, telemetry, and cloud lineage retain the validated remote URI. Temporary files are cleaned after the operation.

Local users authenticate with az login; automation uses its existing federated, workload, managed, or service-principal identity. The identity needs Storage Blob Data Reader, plus any required ADLS path ACLs. Dataset-specific tokens, SAS, account keys, connection strings, query strings, fragments, and embedded credentials are unsupported. Storage firewall and private-endpoint connectivity remain runner responsibilities. execution: azd is unchanged and continues to use the dataset declared by eval.yaml.

POC-to-production readiness flow¶

AgentOps does not create a second production gate with a new exit-code contract. Instead, it projects the signals already produced by evals, Doctor, workflow files, Foundry control-plane checks, Azure Monitor, AI Landing Zone readiness, trace-regression manifests, and optional governance artifacts (ASSERT, ACS, red-team evidence indexes) into a release evidence pack:

flowchart LR
    E["eval run"] --> R["results.json/report.md"]
    R --> D["doctor"]
    D --> P["doctor --evidence-pack"]
    T["exported Foundry/App Insights traces"] --> PT["eval promote-traces --apply"]
    PT --> M["trace-regression-manifest.json"]
    M --> P
    W["workflow generate"] --> P
    G["ASSERT / ACS / red-team evidence paths"] --> P
    P --> J["evidence.json/evidence.md"]

evidence.json is the stable machine-readable contract (version: 1).
evidence.md is the PR/release-review summary.
ready, ready_with_warnings, and blocked are projections of existing signals; eval and Doctor exit codes remain unchanged.
Trace promotion is local and review-first. self-similarity labels are useful for drift detection, not human-verified correctness; use --label-mode pending when a person must fill expected answers before the dataset can gate.
Governance artifacts are evidence-only references. AgentOps records path, SHA-256 hash, file size, schema version when available, and ACS checkpoint coverage; it does not execute ASSERT, apply ACS controls, configure Guided Guardrails, or run red-team campaigns.

Analyze / Generate Boundary¶

The analyze commands are intentionally deterministic and local-only. They do not call models, Copilot, Azure, Foundry, GitHub, or Azure DevOps. Their job is to say whether the next deterministic command is safe or whether the user should ask Copilot to apply the matching skill.

When a skill handoff is needed, the analysis output keeps it copy/pasteable: it reports whether AgentOps Copilot skills are installed in the repo, suggests agentops skills install --platform copilot when they are missing, and prints a single prompt to paste into Copilot.

flowchart LR
    EA["eval analyze"] --> ER["eval run"]
    EA --> ES["Copilot skills when setup is ambiguous"]
    ES --> ER
    WA["workflow analyze"] --> WG["workflow generate"]
    WA --> WS["Copilot workflow skill when deployment is project-specific"]
    ER --> WA
    WG --> CI["Deterministic CI/CD"]
    WS --> CI

CLI Commands¶

Command	Purpose
`agentops --version`	Print the installed version
`agentops explain [COMMAND...]`	Long-form, paged manual for any command (top-level dispatcher)
`agentops init`	Idempotent scaffold + setup wizard (the only onboarding command)
`agentops init show`	Inspect resolved config (`agentops.yaml` + local env values)
`agentops init explain`	Long-form `init` manual
`agentops eval analyze`	Inspect eval setup and recommend direct run vs skill-assisted configuration
`agentops eval init`	Initialize Foundry-native eval assets with `azd`
`agentops eval run`	Run an evaluation; the main command
`agentops eval run --baseline <results.json>`	Run an eval and add a baseline comparison section to the report
`agentops eval promote-traces`	Convert local trace exports into reviewable regression dataset rows
`agentops report generate`	Regenerate `report.md` from a `results.json`
`agentops assert run`	Invoke the ASSERT (`assert-ai`) CLI and normalize its results
`agentops redteam run`	Invoke the Foundry / PyRIT AI Red Teaming agent and normalize its results
`agentops prompt pull`	Pull Foundry prompt-agent instructions into a prompt file
`agentops telemetry validate <name>`	Validate a named telemetry import without querying Azure
`agentops telemetry preview <name>`	Query Azure Monitor and print a small dataset preview
`agentops telemetry import <name>`	Import telemetry into the configured JSONL output path
`agentops telemetry dashboard deploy`	Deploy the Foundry operations workbook to Azure Monitor
`agentops telemetry dashboard open`	Open the Foundry operations workbook in the Azure portal
`agentops telemetry dashboard export`	Export the packaged workbook JSON to a local path
`agentops doctor [--evidence-pack]`	Run the AgentOps Doctor and optionally write release evidence
`agentops doctor explain`	Long-form Doctor manual
`agentops cockpit`	Local read-only Cockpit UI (FastAPI) that links out to Foundry
`agentops cockpit deploy`	Preview or deploy the authenticated hosted Cockpit and explicit Observe scope
`agentops workflow analyze`	Inspect a repo and recommend CI/CD stages/deploy mode before generation
`agentops workflow generate`	Generate CI/CD workflows for GitHub Actions or Azure DevOps
`agentops skills install`	(Re)install coding-agent skills (Copilot or Claude)
`agentops mcp serve`	Run AgentOps as an MCP server for code agents

Every command supports --help for a terse description; long-form, paged documentation is accessible through agentops explain (or the local … explain subcommand where one exists).

Exit Code Contract¶

Exit codes are part of the public API. Do not change their meaning.

Code	Meaning
`0`	Execution succeeded and all thresholds passed
`2`	Execution succeeded but one or more thresholds failed
`1`	Runtime or configuration error

User Workspace Structure (`agentops.yaml` + `.agentops/`)¶

The flat 1.0 schema places one config file at the project root and a small directory for datasets, local environment values, run history, and (optionally) skills. agentops init writes AgentOps-owned local Azure values to .agentops/.env by default. Existing or explicitly requested azd workspaces continue to use .azure/<env>/.env.

<project root>/
├── agentops.yaml              # Single source of truth (flat 1.0 schema)
├── .agentops/
│   ├── data/
│   │   └── smoke.jsonl        # Sample dataset (created by `agentops init`)
│   ├── .env                   # Local Azure values; ignored by `.agentops/.gitignore`
│   ├── results/
│   │   ├── 2026-05-06T14-30-22Z/  # Timestamped run (immutable history)
│   │   │   ├── results.json
│   │   │   ├── report.md
│   │   │   └── cloud_evaluation.json   # only when `publish:` was set
│   │   └── latest/                # Mirror of the most recent run
│   └── agent/                 # Doctor history (history.jsonl + report.md)
├── .azure/                    # Optional: existing or explicit azd env folder
│   ├── config.json            # `defaultEnvironment` pointer
│   ├── .gitignore
│   └── dev/
│       └── .env               # AZURE_AI_FOUNDRY_PROJECT_ENDPOINT,
│                              #  APPLICATIONINSIGHTS_CONNECTION_STRING, …
└── .github/skills/            # Coding agent skills (Copilot)
    ├── agentops-config/SKILL.md
    ├── agentops-eval/SKILL.md
    └── ...

The legacy layered layout (.agentops/config.yaml + bundles/ + datasets/*.yaml + run.yaml) no longer exists. The new schema is declared by src/agentops/core/agentops_config.py and rejects any of the legacy top-level keys (target, bundle, execution, output, scenario, backend, run) at parse time with an actionable error.

`agentops.yaml` (flat 1.0 schema)¶

Minimal config¶

The minimum is three lines:

version: 1
agent: my-rag:3
dataset: ./qa.jsonl

That's a complete config. AgentOps:

Resolves agent into one of four target kinds (see below).
Auto-selects evaluators from the dataset row shape (presence of context, tool_calls, tool_definitions).
Applies sensible default thresholds from the evaluator catalog.

Top-level fields¶

Field	Required	Description
`version`	yes	Schema version. Must be `1`.
`agent`	no	Target identifier. See "Target kinds" below. Omit it to run in project-observability-only mode (Doctor, Cockpit, and Observe against the Foundry project with no evaluation target). Required for `eval run`, `prompt pull`, and the azd eval paths.
`dataset`	yes	Relative/absolute JSONL path, or canonical Blob/ADLS Gen2 HTTPS object URL, with one evaluation row per line.
`thresholds`	no	Metric gates such as `">=3"` or `"<=10"`.
`protocol`	no	URL protocol: `responses`, `invocations`, or `http-json`.
`request_field` / `response_field` / `tool_calls_field`	no	Request/response JSON keys or dot-paths.
`headers`	no	Static HTTP headers (dict).
`auth_header_env`	no	Env var name holding a Bearer token.
`evaluators`	no	Escape-hatch list of evaluator names that overrides auto-selection.
`publish`	no	Boolean. With `execution: local`, `true` uploads local metrics to Classic Foundry. With `execution: cloud`, publishing is implicit.
`execution`	no	`local` (default) runs through AgentOps locally. `cloud` runs a Foundry prompt agent server-side through the OpenAI Evals API.
`project_endpoint`	no	Foundry project URL used by Foundry invocation and publishing. Falls back to `AZURE_AI_FOUNDRY_PROJECT_ENDPOINT`.
`dataset_sync`	no	Cloud-evaluation dataset policy: `auto`, `foundry`, or `inline`.

Target kinds¶

classify_agent() resolves agent into one of four kinds based on shape:

Kind	Trigger	Example `agent` value
`foundry_prompt`	`name:version`	`my-rag:3`
`foundry_hosted`	URL on a Foundry domain	`https://contoso.services.ai.azure.com/.../agents/<id>`
`http_json`	Any other HTTPS URL	`https://my-aca-app.eastus2.azurecontainerapps.io/chat`
`model_direct`	`model:<deployment>`	`model:gpt-4o-mini`

The kind drives both invocation strategy (pipeline/invocations.py) and which fields make sense (e.g. protocol is rejected for foundry_prompt and model_direct).

Foundry Prompt Agents are created and versioned by Foundry. Foundry Hosted Agents are deployed endpoints on a Foundry domain. AgentOps evaluates both, but it does not replace the Foundry Toolkit, Foundry SDK, microsoft-foundry skill, or azd paths that create and deploy them.

Examples¶

Foundry prompt agent (RAG evaluators auto-selected from dataset rows):

version: 1
agent: my-rag:3
dataset: .agentops/data/qa.jsonl
thresholds:
  groundedness: ">=3"
  coherence: ">=3"
  avg_latency_seconds: "<=10"
publish: true         # local run, then upload metrics to Classic Foundry

HTTP-deployed agent (LangGraph / ACA / custom REST):

version: 1
agent: https://my-aca-app.eastus2.azurecontainerapps.io/chat
dataset: .agentops/data/qa.jsonl
request_field: message            # default is "message"
response_field: text              # dot-path; default is "text"
auth_header_env: APP_API_TOKEN    # value used as Bearer token

Raw model deployment:

version: 1
agent: model:gpt-4o-mini
dataset: .agentops/data/qa.jsonl
thresholds:
  similarity: ">=4"
  avg_latency_seconds: "<=8"

Run in New Foundry server-side (preview):

version: 1
agent: my-rag:3                   # name:version is required for cloud mode
dataset: .agentops/data/qa.jsonl
execution: cloud
# project_endpoint: "https://<resource>.services.ai.azure.com/api/projects/<p>"

Datasets¶

A dataset is a plain JSONL file. One row per line. No companion YAML.

Required fields:

Field	Type	Notes
`input`	string	The prompt sent to the target.
`expected`	string	Ground-truth response used by reference-based evaluators.

Optional fields drive evaluator auto-selection:

Field	Triggers
`context`	RAG evaluators
`tool_calls` + `tool_definitions`	Tool-use evaluators

Example RAG row:

{
  "input": "What is the refund policy?",
  "expected": "Refunds within 30 days.",
  "context": "Our policy: refunds are available within 30 days."
}

Local JSONL vs Foundry Data/Datasets¶

The JSONL file referenced by dataset: is the AgentOps source of truth. It belongs in your repo, participates in code review, and is what local and CI runs read.

When execution: cloud is used, Foundry executes the eval server-side. By default, AgentOps syncs the local JSONL to a stable Foundry dataset version and passes that dataset id to the OpenAI Evals API as source: file_id. The Foundry dataset name defaults to agentops-<jsonl-stem> and the version defaults to a content-hash prefix, so unchanged data is reused and changed data creates a new version.

The inline compatibility path is still available with dataset_sync.mode: inline. In that mode, Foundry can materialize inline rows as backing project assets named like eval-data-* in Data > Datasets. Those assets are cloud-eval artifacts, not separate hand-authored AgentOps datasets.

AgentOps records this lineage in cloud_evaluation.json:

{
  "dataset": {
    "mode": "foundry",
    "requested_mode": "auto",
    "source_type": "file_id",
    "local_path": ".agentops/data/smoke.jsonl",
    "sha256": "...",
    "foundry_name": "agentops-smoke",
    "foundry_version": "sha256-..."
  }
}

You can pin the Foundry dataset identity or force inline compatibility:

dataset: .agentops/data/smoke.jsonl
dataset_sync:
  mode: auto                  # auto | foundry | inline
  name: agentops-smoke
  version: content-hash

Use dataset_sync.mode: foundry when a pipeline must fail rather than fall back to inline data if dataset upload/reuse fails. Use dataset_sync.mode: inline for quick experiments or environments that do not have permission to create Foundry datasets.

Evaluator auto-selection¶

The catalog is defined in src/agentops/core/evaluators.py. Selection rules (in order):

If evaluators: is set in agentops.yaml, use it verbatim (escape hatch).
Otherwise, start from the quality baseline for the resolved target kind. Prompt/hosted agents use judge-based answer quality (Coherence + Fluency + Similarity + ResponseCompleteness); raw model:<deployment> targets also add exact-reference overlap (F1Score).
If dataset rows include context, add the RAG evaluator set (Groundedness, Relevance, Retrieval, ResponseCompleteness).
If rows include tool_calls + tool_definitions, add the tool-use evaluator set (ToolCallAccuracy, IntentResolution, TaskAdherence, …).
avg_latency_seconds is always included as a runtime metric.

Recommended judge models¶

AI-assisted evaluators use an LLM as a judge. Use instruction-following models like gpt-4o, gpt-4o-mini, gpt-4.1, gpt-4.1-mini. Avoid reasoning models (o1, o3, o4, gpt-5, gpt-5-nano) - they are slower, more expensive, and may not follow the evaluator prompt format.

Set the deployment via env vars before running:

export AZURE_OPENAI_ENDPOINT="https://<account>.openai.azure.com/"
export AZURE_OPENAI_DEPLOYMENT="gpt-4o-mini"

Thresholds¶

Threshold expressions accept:

Form	Meaning
`">=3"`, `">3"`, `"<=10"`, `"<10"`, `"==1"`	Numeric comparison
`"true"` / `"false"`	Boolean expectation (used by safety evaluators)
Raw number `3`	Shorthand for `>=3`

Each row is judged against every applicable threshold. A row passes only if every threshold passes. The run passes only if every row passes (this is the only condition that maps to exit code 0; otherwise 2).

Publishing to Foundry Evaluations¶

execution: decides where the run happens. publish: controls Foundry visibility for local runs. AgentOps always writes results.json and report.md; cloud runs also write cloud_evaluation.json with the Foundry link.

Config	What runs	Foundry visibility	Targets
`execution: local`, `publish: false`	AgentOps invokes target and evaluators locally	None; local artifacts only	Any target
`execution: local`, `publish: true`	AgentOps local run, then metric upload	Classic Foundry Evaluations panel	Any target
`execution: cloud`	Foundry runs agent + evaluators server-side through the OpenAI Evals API	New Foundry Evaluations panel; publish is implicit	Foundry Prompt Agent (`name:version`) or Foundry Hosted Agent URL containing `/agents/<name>/versions/<version>`

Foundry-visible modes:

Require either project_endpoint in agentops.yaml or AZURE_AI_FOUNDRY_PROJECT_ENDPOINT in the environment.
Authenticate with DefaultAzureCredential (passwordless: az login, managed identity, or service principal).
Write cloud_evaluation.json next to results.json containing mode (classic or cloud), evaluation_name, report_url, and (for cloud execution) the eval_id / run_id / terminal status.

The execution: cloud trade-offs (so you can decide consciously):

Foundry-side latency replaces the locally-measured wall-clock latency.
Judges are opaque (Foundry-managed); custom evaluators are skipped.
The dataset is synced to Foundry Data/Datasets by default. Inline compatibility remains available and may show eval-data-* backing assets.
Evaluator runs cost against your Azure OpenAI deployment.
Polling adds ~5 s × N to the total wall-clock time.

For prompt-agent CI pipelines, AgentOps cloud eval is the default gate when the repo needs artifacts, baselines, Doctor readiness, or release evidence. Foundry executes the managed eval; AgentOps enforces thresholds and writes the proof pack. The standalone Microsoft Foundry AI Agent Evaluation GitHub Action or Azure DevOps extension remains useful for platform-native validation outside the AgentOps release-readiness flow.

Implementation lives in src/agentops/pipeline/publisher.py (Classic) and src/agentops/pipeline/cloud_runner.py (New Foundry). Dispatch happens in src/agentops/pipeline/orchestrator.py.

Pre-flight checks¶

agentops doctor and agentops cockpit run a short pre-flight before doing real work. It lives in src/agentops/services/preflight.py and reports all rows at once instead of stopping at the first problem:

Workspace — the target directory is a usable AgentOps workspace.
Azure authentication — DefaultAzureCredential acquires an ARM token within 30 s (process_timeout=30 is set to absorb Windows az.cmd cold starts).
Foundry project — the configured project endpoint is reachable.
Application Insights — a connection string is available, either auto-discovered from Foundry or set through APPLICATIONINSIGHTS_CONNECTION_STRING.

Each check is a single best-effort attempt with no retries, and a failing check never raises into the CLI. The default policy is advisory: warnings print and the command continues. For CI gating, pass --strict-preflight to agentops doctor so any failure exits non-zero. Pass --no-preflight to doctor or cockpit to skip the checks entirely.

Invocation strategies (target kind → wire call)¶

There is no longer a free-form backend: field. The invocation strategy is derived from the target kind resolved by classify_agent():

Target kind	Invocation strategy
`foundry_prompt`	Foundry Agent Service threads/runs API via `AIProjectClient`
`foundry_hosted`	Direct call to the hosted endpoint with the configured `protocol`
`http_json`	POST `{request_field: input, ...}` and extract `response_field` (dot-path)
`model_direct`	Azure OpenAI chat completions via `AIProjectClient.get_openai_client()`

AIProjectClient.get_openai_client() is always called without api_version - passing one explicitly has historically caused 404s in this codebase.

How evaluators and metrics work¶

Evaluator execution is row-first:
each dataset row is evaluated and can produce one or more row scores.
Threshold evaluation is config-driven:
each entry in thresholds: maps an evaluator's score key to a comparison expression
each row receives a verdict per threshold
a row passes only if every applicable threshold passes
run-level threshold status is consolidated from item verdicts.
Metrics have three levels in results.json:
metrics: backend/global metrics (already aggregated)
row_metrics: per-row evaluator outputs (row_index + metric list + optional input/response text)
item_evaluations: per-row threshold verdicts (per evaluator + final row PASS/FAIL)
run_metrics: consolidated execution metrics derived by AgentOps

In short: - evaluator computes score per item - threshold validates expected quality policy per item and per run - AgentOps consolidates visibility for CI and reporting

Consolidated run metrics¶

AgentOps derives consolidated run metrics for each execution in results.json under run_metrics.
Derived by default:
run_pass (1.0 pass, 0.0 fail)
threshold_pass_rate (thresholds_passed / thresholds_count)
items_total
items_passed_all
items_failed_any
items_pass_rate
per-metric aggregates from row data, for example:
- groundedness_avg
- groundedness_stddev
- latency_seconds_avg
- latency_seconds_stddev
accuracy (from row-level exact_match average when available)

Outputs and history¶

Every run writes its artifacts to .agentops/results/<timestamp>/ (immutable history).
AgentOps then refreshes .agentops/results/latest/ with a copy of that run, so latest/ always points at the most recent results.
Pass --output <dir> to skip the default layout and write only to that path (useful for named baselines or CI artifacts).
results.json: normalized, machine-readable result for CI/automation.
report.md: human-readable summary for review.

When you run:

agentops eval run

AgentOps writes to both:

.agentops/results/YYYY-MM-DD_HHMMSS/ (immutable history of that run)
.agentops/results/latest/ (convenient pointer to last run content)

If you pass --output, AgentOps writes to that directory and still updates .agentops/results/latest/ with the newest run content.

Testing¶

Tests live in tests/ and are organized as:

tests/
├── fixtures/
│   ├── fake_eval_runner.py          # Fake backend for integration tests
│   └── fake_adapter.py              # Fake local adapter (stdin/stdout JSON echo)
├── integration/
│   └── test_eval_run_integration.py # End-to-end via local adapter backend
└── unit/
    ├── test_models.py               # Pydantic model validation
    ├── test_reporter.py             # Threshold evaluation + report
    ├── test_yaml_loader.py          # YAML loading + env-var interpolation
    ├── test_foundry_backend.py      # Foundry backend helpers (mocked)
    ├── test_http_backend.py         # HTTP backend helpers
    └── test_initializer.py          # Workspace scaffolding

Run all tests:

python -m pytest tests/ -x -q

Key testing rules: - All Azure SDK calls must be mocked - tests run without Azure credentials. - Tests must assert correct exit codes (0, 1, 2). - Unit tests go in tests/unit/, integration tests in tests/integration/.

Dependencies¶

Declared in pyproject.toml:

Package	Purpose
`typer`	CLI framework
`pydantic` (v2)	Config and results schema validation
`ruamel.yaml`	YAML parsing with env-var interpolation

Runtime Azure dependencies (installed by the user, not declared in pyproject.toml):

Package	Purpose
`azure-ai-projects`	Foundry project client, `get_openai_client()`
`azure-ai-evaluation`	Local evaluator classes (SimilarityEvaluator, etc.)
`azure-identity`	`DefaultAzureCredential` authentication
`openai`	OpenAI Evals API types

Azure SDK dependencies are kept separate so the CLI stays lightweight and tests can run without cloud credentials.

Quick Reference for New Contributors¶

Install in dev mode: uv sync --group dev. dev is a PEP 735 dependency group, not an extra, so pip install -e ".[dev]" does not install it: pip warns about an unknown extra and installs the project without the dev tools. With pip, use pip install -e ".[agent,mcp]" and add test tools yourself. See the uv installation guide.
Run tests: uv run python -m pytest tests/ -x -q. uv sync populates .venv but does not activate it, so a bare python may be the system interpreter. uv run is what CI uses. If you prefer, activate .venv first and then call python directly.
Try it out: uv run agentops init then explore .agentops/
Read the models: core/models.py is the best single file to understand all data structures
Follow the flow: cli/app.py → services/runner.py → backends/ → core/
Keep CLI thin: never put logic in cli/app.py - delegate to services/
Keep core pure: never import Azure SDK in core/ - that belongs in backends/ and services/