Tests & quality checks
Use these checks when contributing to an application component. Behavioral tests exercise expected and denied outcomes. Static quality checks inspect source, types, imports, and exception handling without proving that an Azure deployment is ready.
Commands below run from the named repository's root unless stated otherwise.
Use Python 3.12 and its own virtual environment, as in
Run locally. Development tools are separate from
requirements.txt and are not an additional runtime dependency manifest.
Select the source and protected base
The platform manifest
at your platform release selects the compatible product revisions.
The commands here follow the current contributor workflows in component
repository main, which may contain newer quality-tooling patches.
They do not adopt newer runtime behavior into the umbrella application.
When working on another revision, inspect its own workflow and tool arguments.
How to choose a useful first check
- Before running commands, identify the changed repository and exact source revision. Component source checkouts are for contributing/testing, not an installation prerequisite for deploying the application.
- Select the smallest behavioral test covering the changed boundary: parser and record failure for ingestion; native host/configuration for orchestration; owner denial and package imports for UI.
- Run it offline with synthetic data, then run that repository's broader checks below. Test fixtures own mocked clients; starting ingestion against Azure storage is not a unit-test shortcut.
- Inspect the JUnit/receipt and quality report. Record failures, skips and unperformed live checks. A successful static report is not evidence that indexing, identity grants or model interpretation worked.
For ordinary contributions targeting develop:
git fetch origin develop
$Base = git merge-base HEAD origin/develop
For release-target or explicitly authorized other work, use the actual fetched target's immutable base SHA instead. A protected base supplies the trusted policy and evaluator. Do not substitute candidate HEAD or a convenient base to suppress findings.
Orchestrator
Prepare tools, run behavioral tests, and bind their JUnit results to the quality evaluation:
python -m pip install -r requirements.txt
python -m pip install -r requirements-hosted.txt -r requirements-container-apps.txt -r requirements-tools.txt
python -m pip check
python -m pip install pytest pytest-asyncio pytest-mock jsonschema
python -m pip install -r requirements-quality.txt
python -m pytest -q --junitxml=.artifacts/pytest.xml
python -I -S .github/scripts/check-quality.py --check all --base-ref $Base --test-results .artifacts/pytest.xml --report .artifacts/quality.json
Use -I -S for this repository's checker and aggregate entrypoint.
The maintained test workflow installs both host SDK requirement files and
tool requirements so SDK-contract tests use published types. Runtime images
still install only their selected host. For local server startup, follow the
separate host setup.
These Python options isolate interpreter startup and skip automatic site
initialization. Static tools analyze fresh source-only snapshots with installed
wheel paths exposed without executing .pth startup hooks.
Candidate packages are analyzed, not imported. This is not an OS sandbox.
Executable mypy plugins and custom Import Linter contracts are rejected.
Choose focused tests for the changed native definitions, installed SDK call shapes, scoped configuration and workflow snapshots. Include owner denial, pending-request mismatch, rejection and duplicate/stale human replies. Run the maintained quality-policy check separately when its policy changes:
python -m pytest -q tests/test_quality_policy.py
If validating the retained source frontend CI job, use Node 20 from
frontend/:
npm install --no-audit --no-fund
npm run build
npm run lint is also available locally. There is no npm test script in
that source frontend. Native orchestrator entrypoints do not serve its
dashboard/admin APIs. Do not invent an active native dashboard or test gate.
Native local SDK/contract checks must exercise the actual product entrypoints and published installed types. They do not establish deployed checkpoint durability, platform cross-user isolation, private-network reachability or exactly-once external writes. Record supplemental live evidence separately and never deploy a full landing zone merely to run a unit or SDK proof.
For attachments, test exact MIME/name/Base64/signature admission, the 64 KiB file and 256 KiB serialized-turn budgets, both native input envelopes and owner denial before backend access. These are transport/contract tests, not proof of model PDF/image understanding.
DAX/delegated multimodal host-token binding must be exercised separately and fails closed when unavailable.
The PR workflow's tests job produces same-run JUnit evidence.
Its quality matrix uses the protected target's evaluator, configuration, and
runtime/tool requirements rather than installing candidate runtime dependencies
in static jobs. quality-gate requires actual successful tests, frontend,
and quality results and fresh reports matching the repository, base, head,
and workflow run. Bootstrap diagnostics are not an activated green policy.
Ingestion
Run the contributor commands independently from the ingestion root:
python -m pip install -r requirements.txt
python -m pip install -r requirements-quality.txt
python -m pip install pytest pytest-asyncio
python -m pytest -q tests/test_quality_policy.py tests/test_jobs_runtime.py
python -m pytest tests -q --junitxml=.artifacts/pytest.xml -o junit_family=legacy
python -I .github/scripts/quality-evidence.py --junit .artifacts/pytest.xml --base-ref $Base --report .artifacts/test-evidence.json
python -I .github/scripts/check-quality.py --check all --base-ref $Base --test-evidence .artifacts/test-evidence.json --report .artifacts/quality.json
Use python -I for its checker, evidence, and aggregate entrypoints.
--repository can select a separate candidate checkout for the protected-base
evaluator. Ordinary contributors can omit it.
The evidence report binds exception and dynamic-import checks to executed tests.
Keep junit_family=legacy when generating that evidence.
Useful focused tests include test_ingest_documents_failures.py,
test_search_write_failures.py, test_chunking_failure_boundaries.py,
test_admin_jobs_queue.py, test_admin_config.py, test_retrieval.py,
and test_panel_operator_auth.py under tests/.
These mock service boundaries. Do not start live scheduled jobs as a shortcut
for an offline test.
Use Node 22. From frontend/:
npm ci
npm test
npm run lint
npm run build
Do not bypass peer validation. These are the maintained frontend-checks
commands, even though the Dockerfile names a Node 20 build stage.
The build produces static/ for the backend, not a deployment.
The test workflow has unit-tests, frontend-checks, and five quality checks.
quality-gate requires successful same-workflow behavior/frontend jobs plus
fresh matching reports. The protected target owns evaluator and tool pins.
A target without an evaluator deliberately blocks a normal green bootstrap
claim.
UI
The inline-visual integration test uses the real orchestrator source with mocked model and Azure calls. Check out the qualified source next to the UI checkout and enable the test before running the suite. In PowerShell:
git clone https://github.com/Azure/agent-app-orchestrator.git ..\agent-app-orchestrator-tests
git -C ..\agent-app-orchestrator-tests checkout 72909db145ca41b954e18261d89e50318bed592a
$env:INLINE_ORCH_TESTS = "1"
$env:INLINE_ORCH_TEST_ROOT = (Resolve-Path ..\agent-app-orchestrator-tests).Path
python -m venv .artifacts\orchestrator-test-env
$env:INLINE_ORCH_TEST_PYTHON = (Resolve-Path .artifacts\orchestrator-test-env\Scripts\python.exe).Path
& $env:INLINE_ORCH_TEST_PYTHON -m pip install -r ..\agent-app-orchestrator-tests\requirements-hosted.txt -r tests\requirements-orchestrator.txt
& $env:INLINE_ORCH_TEST_PYTHON -m pip check
In another shell, export the same three environment variables using absolute paths. CI checks out that exact commit automatically. The UI and orchestrator run in separate test environments and exchange serialized native responses. These sources are test inputs, not part of the installed UI package or production image.
Install the runtime requirements, contributor tools, and editable package:
python -m pip install -r requirements.txt -r requirements-test.txt
python -m pip install -r requirements-quality.txt
python -m pip check
python -m pip install --no-deps -e .
python -m playwright install chromium
python -m unittest discover -s tests -v
On Linux, use python -m playwright install --with-deps chromium to install
the browser's system dependencies too. Playwright and Chromium are test tools,
not production dependencies. Browser tests use synthetic data and run offline
after these tools are installed.
The UI uses standard unittest, not a replacement pytest discovery command.
Focused package acceptance:
python -m pip wheel --no-deps --wheel-dir .artifacts/wheel .
python -m unittest discover -s tests -p test_module_compatibility.py -v
python -m unittest discover -s tests -p test_installed_package.py -v
python -m unittest discover -s tests -p test_quality_policy.py -v
Installed-package acceptance creates a clean non-editable environment, installs
runtime and test requirements, runs pip check, and verifies canonical and
compatibility import origins outside the checkout. It copies tests, not runtime sources.
This can install dependencies and needs package-feed access. It does not prove
a Linux container or Azure integration.
For protected quality checks, create one run ID and use the evidence runner:
$env:QUALITY_RUN_ID = [guid]::NewGuid().ToString()
python .github/scripts/run-unittest.py --base-ref $Base --report .artifacts/unittest.json
python .github/scripts/check-quality.py --check all --base-ref $Base --test-evidence .artifacts/unittest.json --report .artifacts/quality.json
Set QUALITY_RUN_ID once per local run in other shells too.
The receipt binds exact test methods, source fingerprints, base/head, and run
context. Local --pattern supports focused execution.
CI aggregation requires full test_*.py discovery.
Skipped, expected-failure, stale, and failed evidence cannot validate an
exception site. Running the checker without this receipt is not a complete
exception/dynamic-import check.
For protected-source recovery and downloads:
python -m unittest discover -s tests -p test_auth_recovery.py -v
python -m unittest discover -s tests -p test_native_downloads.py -v
tests/auth_recovery_browser.html exercises the bundled helper with synthetic
MSAL and fetch boundaries and makes no application Azure requests. It does not
prove live Entra Conditional Access, browser, or proxy behavior.
CI evaluates quality in a separate environment using protected requirements
and tool pins without installing candidate application code there.
quality-gate requires real unit-test, quality-matrix, and container-tests
results plus fresh bound reports and the unittest receipt.
container-tests
builds the production image, adds browser tools in an ephemeral test image,
and runs tests/container_smoke.py with no
network and read-only mounted UI tests and qualified orchestrator sources.
It checks installed imports, staged
resources, startup, and offline behavior. It publishes and deploys no image.
Quality policy and evidence
All three checkers support all, lint, typing, architecture, exceptions,
and policy. Exit 0 means passed, 1 means violations, and 2 means invalid
inputs or incomplete execution. Missing reports and nonzero results are not
success. Checkers create reports, not source fixes or rewritten baselines.
The policies inventory runtime modules, import directions, public/private surfaces, type coverage, dynamic imports, and broad exception handlers. New runtime modules enter the applicable quality scope automatically. Imported out-of-scope type diagnostics can remain visible without being blocking. Keep the distinction between those diagnostics and enforced scope.
- Orchestrator's protected
module-surfaces.jsonrecords module ownership and public exports. Its initial debt baseline is empty and findings are matched by source, symbol, diagnostic, and multiplicity, not a total count. - Ingestion's initial blocking type scope includes
telemetry/audit_contract.py,telemetry/audit_sanitizer.py, andjobs/runtime.py. - UI's
.quality/migration.jsonpreserves supported compatibility exports while.quality/policy.jsontracks canonical modules and adapters.
An exception record is an exact, bounded allowance, not a blanket waiver for
a category of handlers. Preserve source/handler fingerprints, protected-base
metadata, expiry, stage, and passing same-run evidence as required by the
owning checker. UI active records are eligible for technical validation, not
human approval.
Logging and rethrowing do not automatically waive a handler check. Do not grow baselines, add suppressions, weaken policy, or fabricate evidence to turn a candidate green. Protected checker/tool/workflow changes require explicit maintainer authorization.
Missing, skipped, cancelled, failed, stale, or mismatched CI results do not
authorize a merge. The shared repository standard requires PRs and successful
quality and behavior/frontend or container checks on both develop and main.
The required approving-review count is zero. Code-owner and latest-head approvals are not mandatory, and maintainers may merge their own passing PRs. An authorized administrative exception is temporary, not a permanent bypass or a successful failed check.
Workflow YAML and local commands do not activate or certify live branch
protection. Administrators verify those settings separately.
Unprivileged quality workflows and the separate pull_request_target
main-branch naming guard have different purposes.
For engineering-asset changes, also run the owning repository's
python .github/scripts/validate-agentic-assets.py.
What local checks do not prove
Keep a handoff with the exact source/base, commands, results, and unperformed checks. Offline tests do not certify Azure role assignments, source ACLs, delegated token passthrough, model availability, private DNS, successful indexing, or cross-component readiness.
Test API payloads, headers, bounded retries/timeouts, cancellation, record-level failures, and persistence acknowledgements. Security changes need negative tests for expired, unauthorized, cross-session, cross-origin, and cross-user access. Presentation changes need light/dark, keyboard/focus, responsive, and authenticated/anonymous checks.
Any live validation is a separately authorized task. Follow Sign-in & access, Conversations & files, Jobs & schedules, and Monitoring & audit for their operational boundaries. Code reverts do not restore deleted documents or reverse persisted settings. Data recovery and reingestion require their own plan.
Optional workload evaluation
Consider the AgentOps Accelerator to evaluate, ship, and operate your Agent Landing Zone workload. Linking to AgentOps does not integrate it automatically or enable an evaluation or pull-request gate.