AgentOps Accelerator¶

What AgentOps does¶

AgentOps turns Foundry evaluation, safety, and observability signals into a repeatable ship or no-ship workflow. It connects Foundry Evaluations, the ASSERT safety framework, the PyRIT-backed AI Red Teaming agent, Azure Monitor, and your CI/CD platform into one release loop. Every result is packaged into a stable evidence pack that proves a release is ready for production.

Install in 60 seconds

Install the package, bootstrap a workspace, and drop the skills into your coding agent.

pip install agentops-accelerator
agentops init
agentops skills install --platform copilot

Evaluate¶

Read Evaluation to learn how datasets, evaluators, thresholds, and rubrics turn an agent into a pass or fail gate. Start with agentops eval run and the agentops-eval skill.

Ship¶

Ship explains the generated PR gate and dev deploy workflows, and how candidate versions become a release. Start with agentops workflow generate --kinds pr and the agentops-workflow skill.

Observe¶

Observe covers Foundry traces and Azure Monitor, and how production signals feed continuous evaluation. Start with agentops telemetry validate and the agentops-agent skill.

Operate¶

Operate shows how Doctor scores readiness and packages an evidence pack so you can make the ship or no-ship call. Start with agentops doctor and the agentops-governance skill.

Reference architecture¶

Use this as the mental model for the AgentOps loop: build in a sandbox, commit the release contract to source control, promote through environments with evidence, then feed production learning back into the next evaluation set.

AgentOps Accelerator reference architecture

Where to go next¶

Pick a tutorial to learn the sandbox to PR gate flow end to end, or jump straight to the evaluation reference.

Tutorials Evaluation reference

Contributions are welcome. See the repository for guidelines, issues, and the contribution process.