Microsoft Foundry

---
name: microsoft-foundry
description: "Deploy, evaluate, fine-tune, and manage Foundry agents end-to-end with azd: hosted agent scaffold/run/deploy, prompt agent create, batch eval, continuous eval, prompt optimizer, Agent Optimizer scaffold, agent.yaml, dataset curation from traces, model fine-tuning (SFT/DPO/RFT). USE FOR: azd ai agent, azd provision/deploy, deploy agent, hosted agent, create agent, add tool to agent, invoke agent, evaluate agent, continuous eval, continuous monitoring, optimize prompt, improve prompt, optimize agent instructions, agent optimizer, deploy model, Foundry project, RBAC, role assignment, permissions, quota, capacity, region, troubleshoot agent, deployment failure, AI Services, create Foundry resource, provision, knowledge index, customize deployment, onboard, availability, fine-tune, SFT, DPO, RFT, training-data, grader, distillation, fine-tuned model, large file upload. DO NOT USE FOR: Azure Functions, App Service, general Azure deploy (use azure-deploy), general Azure prep (use azure-prepare)."
license: MIT
metadata:
  author: Microsoft
  version: "1.1.37"
---

# Microsoft Foundry Skill

This skill helps developers work with Microsoft Foundry resources, covering model discovery and deployment, complete dev lifecycle of AI agent, evaluation workflows, and troubleshooting.

## Pre-Execution Requirements

Before using Foundry MCP operations, call the Azure MCP `foundry` tool and inspect the available Foundry MCP tools and related parameters. Treat this as the discovery/help step for MCP-based workflows.

## Sub-Skills

> **MANDATORY: Before executing ANY workflow-specific steps, you MUST read the corresponding sub-skill document.** Do not call workflow-specific MCP tools for a workflow without reading its skill document. This applies even if you already know the MCP tool parameters — the skill document contains required workflow steps, pre-checks, and validation logic that must be followed. This rule applies on every new user message that triggers a different workflow, even if the skill is already loaded.

This skill includes specialized sub-skills for specific workflows. **Use these instead of the main skill when they match your task:**

| Sub-Skill | When to Use | Reference |
|-----------|-------------|-----------|
| **deploy** | Deploy hosted agents to Foundry, smoke-test a deployment, create or update prompt agents, and manage agent versions and multi-environment deploys. | [deploy](foundry-agent/deploy/deploy.md) |
| **invoke** | Send messages to an agent, single or multi-turn conversations | [invoke](foundry-agent/invoke/invoke.md) |
| **routine** | Schedule or event-trigger Foundry agents with routines; use `azd` for CRUD, enable/disable, manual dispatch, and viewing past runs, or define routines in `azure.yaml`. | [routine](foundry-agent/routine/routine.md) |
| **invocations-ws** | Build, deploy, and connect to hosted agents that speak the `invocations_ws` duplex WebSocket protocol — voice agents, real-time streams, and signaling for out-of-band media transports. | [invocations-ws](foundry-agent/invocations-ws/invocations-ws.md) |
| **observe** | Evaluate agent quality, run batch evals, analyze failures, optimize prompts, improve agent instructions, compare versions, set up CI/CD monitoring, and enable continuous production evaluation | [observe](foundry-agent/observe/observe.md) |
| **trace** | Query traces, analyze latency/failures, correlate eval results to specific responses via App Insights `customEvents` | [trace](foundry-agent/trace/trace.md) |
| **troubleshoot** | View hosted agent logs, query telemetry, diagnose failures | [troubleshoot](foundry-agent/troubleshoot/troubleshoot.md) |
| **create (quick start)** | Create a new hosted Foundry agent from scratch end-to-end — scaffold, provision or use an existing Foundry project, deploy, and smoke-test. Opinionated happy-path that accepts common overrides (language, region, sample, topic, existing project, existing model). For anything not covered by the quickstart, use **create**. | [create/quick-start-hosted.md](foundry-agent/create/quick-start-hosted.md) |
| **create** | Use when the standard end-to-end happy path doesn't fit — lifting existing agent code into the project, deploying outside the default code path, wiring connections at scaffold time, advanced setup, or recovering from a failed quickstart run. | [create](foundry-agent/create/create-hosted.md) |
| **agent-optimizer** | Make existing Python hosted-agent code optimization-ready, configure eval.yaml, run Agent Optimizer jobs, apply candidates locally, and deploy through azd after review. | [agent-optimizer](foundry-agent/agent-optimizer/agent-optimizer.md) |
| **eval-datasets** | Harvest production traces into evaluation datasets, manage dataset versions and splits, track evaluation metrics over time, detect regressions, and maintain full lineage from trace to deployment. Use for: create dataset from traces, dataset versioning, evaluation trending, regression detection, dataset comparison, eval lineage. | [eval-datasets](foundry-agent/eval-datasets/eval-datasets.md) |
| **project/create** | Creating a new Azure AI Foundry project for hosting agents and models. Use when onboarding to Foundry or setting up new infrastructure. | [project/create/create-foundry-project.md](project/create/create-foundry-project.md) |
| **resource/create** | Creating Azure AI Services multi-service resource (Foundry resource) using Azure CLI. Use when manually provisioning AI Services resources with granular control. | [resource/create/create-foundry-resource.md](resource/create/create-foundry-resource.md) |
| **private-network** | Answer questions about Foundry network isolation **and** deploy Foundry with VNet isolation (BYO VNet, Managed VNet, hybrid). Covers architecture concepts, template selection, deployment, and post-deployment validation. | [resource/private-network/private-network.md](resource/private-network/private-network.md) |
| **models/deploy-model** | Unified model deployment with intelligent routing. Handles quick preset deployments, fully customized deployments (version/SKU/capacity/RAI), and capacity discovery across regions. Routes to sub-skills: `preset` (quick deploy), `customize` (full control), `capacity` (find availability). | [models/deploy-model/SKILL.md](models/deploy-model/SKILL.md) |
| **quota** | Managing quotas and capacity for Microsoft Foundry resources. Use when checking quota usage, troubleshooting deployment failures due to insufficient quota, requesting quota increases, or planning capacity. | [quota/quota.md](quota/quota.md) |
| **rbac** | Managing RBAC permissions, role assignments, managed identities, and service principals for Microsoft Foundry resources. Use for access control, auditing permissions, and CI/CD setup. | [rbac/rbac.md](rbac/rbac.md) |
| **finetuning** | Fine-tune models on Azure AI Foundry — SFT distillation, DPO preference optimization, RFT with graders and tool calling. Dataset preparation, grader calibration, training, checkpoint selection, deployment, evaluation. Use for: fine-tune, SFT, DPO, RFT, training data, grader, distillation, fine-tuned model, large file upload. | [finetuning/SKILL.md](finetuning/SKILL.md) |

> 💡 **Tip:** For a complete onboarding flow: `project/create` (public) or `private-network` (VNet isolation) → `models/deploy-model` → agent workflows (`create` → `deploy` → `invoke`).

> 💡 **Fine-Tuning:** Use `finetuning` for all model customization — SFT distillation, DPO preference optimization, and RFT with graders. Includes quickstart, grader calibration, and training curve analysis.

> 💡 **Model Deployment:** Use `models/deploy-model` for all deployment scenarios — it intelligently routes between quick preset deployment, customized deployment with full control, and capacity discovery across regions.

> 💡 **Prompt Optimization:** For requests like "optimize my prompt" or "improve my agent instructions," load [observe](foundry-agent/observe/observe.md) and use the `prompt_optimize` MCP tool through that eval-driven workflow.

## Infrastructure Lifecycle

Match user intent to the correct infrastructure workflow.

| User Intent | Workflow |
|-------------|---------|
| "Create Foundry" / "Set up Foundry" (ambiguous) | Use `AskUserQuestion`: (a) just an AI Services resource, (b) a project with public access, or (c) a project with network isolation? Route: (a) → [resource/create](resource/create/create-foundry-resource.md), (b) → [project/create](project/create/create-foundry-project.md), (c) → [private-network](resource/private-network/private-network.md) |
| Set up Foundry with VNet isolation | [private-network](resource/private-network/private-network.md) |
| Create a Foundry project (public) | [project/create](project/create/create-foundry-project.md) |
| Create a bare Foundry resource | [resource/create](resource/create/create-foundry-resource.md) |

## Agent Development Lifecycle

Match user intent to the correct agent workflow. Read each sub-skill in order before executing.

| User Intent | Workflow (read in order) |
|-------------|------------------------|
| Create a new hosted agent end-to-end (scaffold + deploy + test) | [quick-start-hosted](foundry-agent/create/quick-start-hosted.md) (self-contained end-to-end) |
| Anything beyond the standard quickstart (existing code, deployment customization, scaffold-time connections, recovery) | [create](foundry-agent/create/create-hosted.md) → [deploy](foundry-agent/deploy/deploy.md) → [invoke](foundry-agent/invoke/invoke.md) |
| Optimize existing Python hosted agent | [agent-optimizer](foundry-agent/agent-optimizer/agent-optimizer.md) → scaffold/review → eval.yaml → optimize → apply candidate → deploy → invoke |
| Deploy an agent (code already exists) | deploy (includes eval-suite setup) → invoke → observe (evaluate/optimize) |
| Update/redeploy an agent after code changes | deploy (includes eval-suite setup) → invoke → observe (evaluate/optimize) |
| Invoke/test/chat with an agent | invoke |
| Schedule/event-trigger an agent, or CRUD/enable/disable/dispatch a routine | routine |
| Optimize / improve agent prompt or instructions | observe (Step 4: Optimize) |
| Evaluate and optimize agent (full loop) | observe |
| Enable continuous evaluation monitoring | observe (Step 6: CI/CD & Monitoring) |
| Troubleshoot an agent issue | invoke → troubleshoot |
| Fix a broken agent (troubleshoot + redeploy) | invoke → troubleshoot → apply fixes → deploy → invoke |

## Agent: .foundry Workspace Standard

Every agent source folder can keep Foundry-specific cache and overlay state under `.foundry/`:

```text
<agent-root>/
  .foundry/
    agent-metadata.yaml
    agent-metadata.prod.yaml
    suites/
    datasets/
    evaluators/
    results/
```

- In azd projects, derive deployment context (project endpoint, agent name/version, ACR, App Insights) from `azure.yaml` plus `azd env get-values`; do not duplicate those values in metadata when azd already provides them.
- `agent-metadata.yaml` is the preferred local/dev overlay for non-azd values, remote Foundry suite references, local cache paths, result summaries, and explicit overrides. Optional sidecar files such as `agent-metadata.prod.yaml` can hold a single prod or CI-targeted overlay without mixing multiple environments in one file.
- `suites/`, `datasets/`, and `evaluators/` are local cache folders. Reuse them when they are current, and ask before refreshing or overwriting them.
- See [Agent Metadata Contract](references/agent-metadata-contract.md) for the canonical schema and workflow rules.

## Agent: Setup References

- [Standard Agent Setup](references/standard-agent-setup.md) — advanced setup for production workloads that need data-residency control (bring-your-own Cosmos DB / Storage / AI Search via a Foundry capability host). The default `azd ai agent` flow uses **Basic Agent Setup** and does **not** provision `capabilityHosts/agents` — do not flag its absence as a bug. For default post-provision state, see the "Expected env-var fingerprint" section in [foundry-agent/create/create-hosted.md](foundry-agent/create/create-hosted.md).

## Agent: Common Project Context Resolution

Agent skills should run this step **only when they need configuration values they don't already have**. If a value (for example, agent root, environment, project endpoint, or agent name) is already known from the user's message or a previous skill in the same session, skip resolution for that value.

### Step 1: Discover Agent Roots and azd Context

First check whether the workspace has `azure.yaml` with services using `host: azure.ai.agent`.

- **One azd agent service** -> use that service's `project` folder as the agent root.
- **Multiple azd agent services** -> require the user to choose the target service/folder.
- **No azd agent service** -> search the workspace for `.foundry/` folders that contain `agent-metadata.yaml` or `agent-metadata.<env>.yaml`.
  - **One match** -> use that agent root.
  - **Multiple matches** -> require the user to choose the target agent folder.
  - **No matches** -> for create/deploy workflows, seed a new `.foundry/` folder during setup; for all other workflows, stop and ask the user which agent source folder to initialize.

After selecting an agent root, keep all local `.foundry` cache inspection, source inspection, evaluator suggestions, dataset suggestions, and prompt-optimization context inside that folder only. Do **not** scan sibling agent folders unless the user explicitly switches roots.

### Step 2: Resolve Environment and Deployment Context

If `azure.yaml` is present, resolve the azd environment first:

1. Environment explicitly named by the user
2. `AZURE_ENV_NAME` from `azd env get-values`
3. azd default environment from `.azure/config.json`
4. Environment already selected earlier in the session

Run `azd env get-values` for the selected environment when project/deployment values are not already known. Prefer azd values for deployment context:

| azd Variable | Resolves To |
|-------------|-------------|
| `AZURE_AI_PROJECT_ENDPOINT` or `AZURE_AIPROJECT_ENDPOINT` | Project endpoint |
| `AGENT_<SERVICE>_NAME` | Agent name for the selected azd service |
| `AGENT_<SERVICE>_VERSION` | Agent version for the selected azd service |
| `AZURE_CONTAINER_REGISTRY_NAME` or `AZURE_CONTAINER_REGISTRY_ENDPOINT` | ACR registry name / image URL prefix |
| `APPLICATIONINSIGHTS_CONNECTION_STRING` | App Insights connection string for trace workflows |
| `AZURE_SUBSCRIPTION_ID`, `AZURE_RESOURCE_GROUP`, `AZURE_AI_ACCOUNT_NAME`, `AZURE_AI_PROJECT_NAME` | Azure resource lookup and Playground links |

When azd supplies these values, use them as the source of truth and do not copy them into `.foundry/agent-metadata*.yaml` on metadata writes.

### Step 3: Select Metadata Overlay and Resolve Environment

Inside the selected agent root, choose the metadata file in this order:
1. Metadata filename or path explicitly provided by the user or workflow
2. If an explicit environment is already known and `.foundry/agent-metadata.<env>.yaml` exists, use that file
3. `.foundry/agent-metadata.yaml`
4. If multiple metadata files remain and no rule above selects one, prompt the user to choose

Read the selected metadata file and resolve any remaining environment choice in this order:
1. Environment explicitly named by the user
2. If the selected metadata file defines exactly one environment, use it
3. Environment already selected earlier in the session
4. `defaultEnvironment` from metadata

If the selected metadata file still contains multiple environments and none of the rules above selects one, prompt the user to choose. Keep the selected agent root, metadata file, environment, and whether context came from azd or metadata visible in every workflow summary.

If the selected environment exposes older `testSuites[]` metadata but not `evaluationSuites[]`, treat `testSuites[]` as the source for this session and normalize each entry in memory to the `evaluationSuites[]` shape before continuing. If the metadata is older still and only exposes legacy `testCases[]`, normalize that list the same way. Preserve dataset and evaluator fields, keep any existing `tags`, and map legacy `priority` to `tags.tier` only when `tags.tier` is missing: `P0` -> `smoke`, `P1` -> `regression`, `P2` -> `coverage`.

### Step 4: Resolve eval.yaml Local Evaluation Intent

If `eval.yaml` exists in the selected agent root, parse it before generating new suites:

- `agent.name` -> target agent candidate; verify it matches the selected azd/metadata agent before using it.
- `dataset.local_uri` -> local seed dataset candidate; legacy `dataset_file` may be normalized in memory.
- `dataset.name` / `dataset.version` -> registered dataset candidate.
- `validation_dataset` -> optional validation dataset candidate.
- `evaluators[]` -> candidate Foundry evaluator names; verify with `evaluator_catalog_get` before treating them as remote evaluators.
- `name` -> local eval/suite candidate; verify remotely before persisting as `suiteName`.
- `options.eval_model`, `options.optimization_model`, `options.max_candidates`, `options.optimization_config.model_search_space`, `options.pass_threshold`, `max_samples`, `trace_days`, and `generation_instruction` -> setup defaults.

Treat `eval.yaml` as local evaluation intent, not proof that a Foundry suite exists. Persist synced suite/dataset/evaluator references to `.foundry` only after remote lookup or registration succeeds.

### Step 5: Resolve Common Configuration

Layer sources in this order:

1. Explicit user input and values already selected in the session
2. azd environment values for deployment context
3. `.foundry/agent-metadata*.yaml` overlay values and remote suite/cache references
4. `azure.yaml` and `eval.yaml` local source configuration
5. User prompts for anything still missing

If azd and metadata both provide the same value and they differ, stop and ask which source is authoritative. If they match, use the azd value and avoid rewriting the duplicate on future metadata writes.

| Effective Value | Preferred Source | Used By |
|-----------------|------------------|---------|
| Project endpoint | azd env | deploy, invoke, observe, trace, troubleshoot |
| Agent name/version | azd agent variables, then `azure.yaml` | invoke, observe, trace, troubleshoot |
| ACR | azd env | deploy |
| Evaluation suites and cache paths | `.foundry/agent-metadata*.yaml` | observe, eval-datasets |
| Local seed dataset/evaluator intent | `eval.yaml` | observe, eval-datasets |

### Step 6: Write Metadata Overlay (Create/Deploy/Observe Only)

On any metadata write (deploy, auto-setup, dataset refresh, or trace-to-dataset update), persist only non-derivable overlay/cache state in the selected metadata file:

- azd binding (`azd.environmentName`, `azd.service`) when useful for future resolution
- `evaluationSuites[]` with remote suite/dataset/evaluator references and local cache paths
- `lastEval`, result files, comparison summaries, or explicit non-azd overrides

Do not copy azd-owned deployment values into metadata when azd already provides them. If the selected file is a preferred single-environment file, rewrite only that one environment block. If the selected file is a legacy multi-environment file, rewrite only the selected environment block. Never copy or merge environments across sibling metadata files automatically. If the selected environment still uses older `testSuites[]` or legacy `testCases[]`, rewrite it to `evaluationSuites[]` and remove migrated `priority` fields from the rewritten entries.

### Step 7: Collect Missing Values

Use the `ask_user` or `askQuestions` tool **only for values not resolved** from the user's message, session context, metadata, or azd bootstrap. Common values skills may need:
- **Agent root** — Target azd service project folder or folder containing `.foundry/agent-metadata*.yaml`
- **Metadata file** — `agent-metadata.yaml` for local/dev, or an explicit sidecar such as `agent-metadata.prod.yaml`
- **Environment** — azd environment, `dev`, `prod`, or another environment key from metadata
- **Project endpoint** — AI Foundry project endpoint URL
- **Agent name** — Name of the target agent

> 💡 **Tip:** If the user already provides the agent path, environment, project endpoint, or agent name, extract it directly — do not ask again.

## Agent: Agent Types

All agent skills support two agent types:

| Type | Kind | Description |
|------|------|-------------|
| **Prompt** | `"prompt"` | LLM-based agents backed by a model deployment |
| **Hosted** | `"hosted"` | Container-based agents running custom code |

Use `agent_get` MCP tool to determine an agent's type when needed.

## Tool Usage Conventions

- Use the `ask_user` or `askQuestions` tool whenever collecting information from the user
- Use the `task` or `runSubagent` tool to delegate long-running or independent sub-tasks (e.g., env var scanning, status polling, Dockerfile generation)
- Prefer Azure MCP tools over direct CLI commands when available
- Reference official Microsoft documentation URLs instead of embedding CLI command syntax

## Additional Resources

- [Foundry Hosted Agents](https://learn.microsoft.com/azure/ai-foundry/agents/concepts/hosted-agents?view=foundry)
- [Foundry Agent Runtime Components](https://learn.microsoft.com/azure/ai-foundry/agents/concepts/runtime-components?view=foundry)

## SDK Quick Reference

- [Python](references/sdk/foundry-sdk-py.md)

## Network Isolation Errors

Applies to **any** call against a Foundry project or its parent Foundry account — Foundry MCP tools, `azd`, `az` CLI, `curl`, REST, or SDK.

If an error matches `Public access is disabled` / `PublicNetworkAccessDisabled` / `403 Forbidden` from a private endpoint / connection timeout / the project endpoint FQDN resolves to a public IP, this typically means the parent Foundry account has `publicNetworkAccess=Disabled` or `Enabled from selected IP addresses`, and the current shell is outside its VNet.

Only if the error is ambiguous, confirm against the Foundry account using a management-plane call (works from anywhere with reader access):

```bash
az cognitiveservices account show \
  --name <account> --resource-group <rg> \
  --query "properties.{publicNetworkAccess:publicNetworkAccess, networkAcls:networkAcls, privateEndpointConnections:privateEndpointConnections[].properties.privateLinkServiceConnectionState.status}"
```

`publicNetworkAccess: "Disabled"` — or `"Enabled"` together with non-empty `networkAcls.ipRules` / `virtualNetworkRules` — confirms isolation. If `publicNetworkAccess: "Enabled"` and `networkAcls` is empty, the failure is a caller-side network issue (e.g. Private DNS resolving the FQDN to a public IP from inside a VNet with a private endpoint), not an account-config issue.

If it's indeed a network isolation issue, supported connection options are documented in [Choose a secure connection method to Foundry](https://learn.microsoft.com/azure/foundry/how-to/configure-private-link#choose-a-secure-connection-method-to-foundry).

> ℹ️ Foundry MCP tools cannot reach a VNet-isolated project even from inside the VNet.

---
name: finetuning
description: "Fine-tune models on Azure AI Foundry using SFT (supervised), DPO (preference), or RFT (reinforcement with graders). Covers dataset preparation, training job submission, deployment, and evaluation. USE FOR: fine-tune, SFT, DPO, RFT, training data, grader, distillation, fine-tuned model, training job, large file upload, calibrate grader, deploy fine-tuned model, evaluate fine-tuned model. DO NOT USE FOR: general model deployment without fine-tuning (use deploy-model), agent creation (use agents), prompt optimization without training (use prompt-optimizer)."
license: MIT
metadata:
  author: Microsoft
  version: "0.0.0-placeholder"
---

# Fine-Tuning on Azure AI Foundry

Fine-tune models using SFT (supervised), DPO (preference), or RFT (reinforcement with graders). Covers dataset prep, training, deployment, and evaluation.

## When to Use

Use this sub-skill when the user asks about:
- Fine-tuning a model (SFT, DPO, or RFT)
- Preparing, validating, or formatting training data
- Submitting, monitoring, or diagnosing training jobs
- Calibrating graders or pass thresholds for RFT
- Deploying or evaluating a fine-tuned model
- Choosing between training types (SFT vs DPO vs RFT)
- Distillation, synthetic data generation, or dataset quality scoring
- Large file uploads for training data
- Cleaning up fine-tuning resources (files, deployments)

**Do NOT use for:** General model deployment without fine-tuning (use deploy-model), agent creation (use agents), prompt optimization without training (use prompt-optimizer).

## Workflows

| Stage | Guide |
|-------|-------|
| **Quick start** | [workflows/quickstart.md](workflows/quickstart.md) |
| **Full pipeline** | [workflows/full-pipeline.md](workflows/full-pipeline.md) |
| **Create data** | [workflows/dataset-creation.md](workflows/dataset-creation.md) |
| **Iterate** | [workflows/iterative-training.md](workflows/iterative-training.md) |
| **Diagnose** | [workflows/diagnose-poor-results.md](workflows/diagnose-poor-results.md) |

## References

| Topic | File |
|-------|------|
| SFT vs DPO vs RFT | [references/training-types.md](references/training-types.md) |
| Hyperparameters | [references/hyperparameters.md](references/hyperparameters.md) |
| Data formats | [references/dataset-formats.md](references/dataset-formats.md) |
| Grader design (RFT) | [references/grader-design.md](references/grader-design.md) |
| Reward hacking | [references/reward-hacking.md](references/reward-hacking.md) |
| Agentic RFT (tools) | [references/agentic-rft.md](references/agentic-rft.md) |
| Deployment | [references/deployment.md](references/deployment.md) |
| Training curves | [references/training-curves.md](references/training-curves.md) |
| Evaluation | [references/evaluation.md](references/evaluation.md) |
| Vision fine-tuning | [references/vision-fine-tuning.md](references/vision-fine-tuning.md) |
| Large file uploads | [references/large-file-uploads.md](references/large-file-uploads.md) |
| Platform gotchas | [references/platform-gotchas.md](references/platform-gotchas.md) |

## Scripts

| Script | Purpose |
|--------|---------|
| `scripts/submit_training.py` | Submit SFT/DPO/RFT jobs |
| `scripts/monitor_training.py` | Poll job until completion |
| `scripts/calibrate_grader.py` | Find optimal RFT pass_threshold |
| `scripts/check_training.py` | Analyze curves, list checkpoints |
| `scripts/deploy_model.py` | Deploy via ARM REST API |
| `scripts/evaluate_model.py` | LLM judge evaluation |
| `scripts/convert_dataset.py` | Convert between SFT/DPO/RFT formats |
| `scripts/generate_distillation_data.py` | Generate synthetic training data |
| `scripts/score_dataset.py` | Quality scoring on training data |
| `scripts/cleanup.py` | Delete old files and deployments |
| `scripts/validate/` | Data validators (SFT, DPO, RFT) + stats |

## Rules

1. **Always baseline first** — evaluate the base model before fine-tuning
2. **Validate data** before submitting — run `scripts/validate/validate_sft.py`
3. **Calibrate RFT graders** — target 25-50% failure rate on the base model
4. **Evaluate checkpoints** — don't blindly deploy the final one
5. **Measure token cost** alongside accuracy when comparing models

## Quick Reference

| Task | Command |
|------|---------|
| Validate SFT data | `python scripts/validate/validate_sft.py data.jsonl` |
| Submit SFT job | `python scripts/submit_training.py --model gpt-4.1-mini --training-file train.jsonl --validation-file val.jsonl --type sft` |
| Monitor job | `python scripts/monitor_training.py --job-id ftjob-xxx` |
| Analyze curves | `python scripts/check_training.py --job-id ftjob-xxx` |
| Deploy model | `python scripts/deploy_model.py --model-id ft:gpt-4.1-mini:... --name my-eval` |
| Evaluate model | `python scripts/evaluate_model.py --deployment-name my-eval --test-file test.jsonl` |

## Error Handling

| Error | Cause | Fix |
|-------|-------|-----|
| "API version not supported" | Older `openai` SDK on `/v1/` endpoint | Upgrade to `openai>=1.0` |
| "does not support fine-tuning with Standard TrainingType" | OSS model needs `globalStandard` | Use `--use-rest` flag or script auto-falls back |
| Job stuck in post-training eval | Under-provisioned tool endpoint (RFT) | Scale to S2+, enable Always On |
| "DeploymentNotReady" after ARM succeeds | ARM/data-plane race condition | Delete and recreate deployment, wait 5 min |
| Content safety block at deployment | PII-dense training data | Remove problematic document types |

# Agentic RFT — Tool Calling

Train reasoning models (o4-mini) for agentic scenarios where the model invokes external tools during chain-of-thought reasoning.

> ⚠️ **Access required**: Agentic RFT with tool calling and GPT-5 RFT are behind feature flags. You must request access through the Azure AI Foundry portal or your Microsoft account team. o4-mini RFT without tools is generally available.

## Tool Definition Format

```python
tools = [
    {
        "name": "search",
        "server_url": "https://your-function-app.azurewebsites.net/api/tools",
        "headers": {
            "Authorization": "Bearer <your-key>"
        }
    },
    {
        "name": "get_by_id",
        "server_url": "https://your-function-app.azurewebsites.net/api/tools",
        "headers": {
            "Authorization": "Bearer <your-key>"
        }
    }
]
```

## Submitting an Agentic RFT Job

```python
job = client.fine_tuning.jobs.create(
    model="o4-mini-2025-04-16",
    training_file=train.id,
    validation_file=valid.id,
    method={
        "type": "reinforcement",
        "reinforcement": {
            "grader": grader,
            "tools": tools,
            "max_episode_steps": 10,
            "hyperparameters": {
                "eval_interval": 5,
                "eval_samples": 10,
                "compute_multiplier": 1.5,
                "reasoning_effort": "medium"
            }
        }
    }
)
```

## Tool Response Format

Your tool endpoint must return:

```json
{
    "type": "function_call_output",
    "call_id": "call_12345xyz",
    "output": "The result of the tool call...",
    "id": "fc_12345xyz"
}
```

## Tool Endpoint Requirements

| Constraint | Limit |
|-----------|-------|
| Recommended throughput | 50 QPS |
| Max input payload | 1 MB |
| Max return payload | 1 MB (413 error if exceeded) |
| Timeout | 10 minutes |
| Parallel calls | Supported — handle race conditions |
| Retry on 5xx | 3 attempts, then rollout discarded |
| On 4xx | Error serialized and shown to model |

**Infrastructure**: Use Always On, sufficient compute (S2+), multiple instances. Under-provisioned endpoints can cause jobs to hang during post-training eval.

## RFT Hyperparameters

| Parameter | Description | Recommended Start |
|-----------|-------------|-------------------|
| `reasoning_effort` | `"low"`, `"medium"`, `"high"` | `"medium"` |
| `compute_multiplier` | Scales rollouts per step | `1.5` |
| `learning_rate_multiplier` | Scales the learning rate | `1.0` |
| `n_epochs` | Data passes | `2–3` |
| `eval_interval` | Eval every N steps | `5` |
| `eval_samples` | Validation examples per eval | `10` |
| `max_episode_steps` | Max tool calls + reasoning steps per rollout | `5–10` |

**Notes:** Higher LR increases output verbosity without improving accuracy. Compute multiplier 1.5 balances rollout quality and training time. Platform may early-stop before all epochs.

## When to Use Agentic RFT

- Model needs to **decide when to call tools** (not just follow instructions)
- Task involves **multi-step reasoning** with external data lookups
- Model needs to learn **tool selection** — choosing the right tool for the job
- Standard RFT (without tools) can't capture the agentic behavior

# Dataset Formats

## SFT Format (Supervised Fine-Tuning)

Standard chat-completion JSONL. Each line: JSON object with `messages` array.

```jsonl
{"messages": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "What is 2+2?"}, {"role": "assistant", "content": "4"}]}
```

**Rules:**
- Each line must be valid JSON
- `messages` must contain at least one `user` and one `assistant` message
- `system` message is optional but recommended
- Multi-turn supported: alternate `user`/`assistant`
- Last message must be `assistant` (that's what the model learns)

**Validation checklist:** `.jsonl` extension, valid JSON per line, every example has `messages`, every message has `role` and `content`, no empty `content`.

## DPO Format (Direct Preference Optimization)

Three top-level fields: `input`, `preferred_output`, `non_preferred_output`.

```jsonl
{"input": {"messages": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Explain gravity."}]}, "preferred_output": [{"role": "assistant", "content": "Gravity is a fundamental force that attracts objects with mass toward each other."}], "non_preferred_output": [{"role": "assistant", "content": "Gravity is when stuff falls down."}]}
```

**Rules:**
- `input`: Object with `messages` array (system + user turns). May include `tools` and `parallel_tool_calls`.
- `preferred_output` / `non_preferred_output`: Array of messages (`assistant` or `tool` role only)
- Both must contain at least one `assistant` message
- Exactly two completions compared per example

**DPO REST API example:**
```json
{
  "model": "gpt-4.1-mini-2025-04-14",
  "training_file": "file-abc123",
  "method": {
    "type": "dpo",
    "dpo": { "beta": 0.1, "l2_multiplier": 0.1 }
  }
}
```

## RFT Format (Reinforcement Fine-Tuning)

Chat-completion format with key differences from SFT:

```jsonl
{"messages": [{"role": "user", "content": "Write a Python function to reverse a string."}], "reference_code": "def reverse_string(s):\n    return s[::-1]", "expected_output": "olleh"}
```

**Rules:**
- Last message **MUST** be `user` role (model generates its own response)
- Extra fields alongside `messages` are accessible to grader via `item.*`
- Both training and validation datasets are **required**
- ⚠️ Do NOT put `assistant` as last message — unlike SFT, RFT generates its own outputs

**API version**: Python graders require `api-version=2025-04-01-preview` or later.

**Grader types:** `string_check` (exact match), `text_similarity` (fuzzy/BLEU/ROUGE), `python` (custom function), `score_model` (LLM judge), `multi` (weighted combination).

**Python grader template:**
```python
def grade(sample, item):
    """
    sample: dict with 'output_text' (model's generation)
    item: dict with extra fields from JSONL
    Returns: float 0.0–1.0
    """
    output = sample.get("output_text", "")
    reference = item.get("reference_code", "")
    return score
```

**Python grader constraints:** 256KB code max, no network, 2GB memory, 1GB disk, 2min timeout.

**Grader field access:**
- `sample.output_text` → model's generation
- `sample.output_json` → structured output (if using response_format)
- `item.*` → extra JSONL fields
- Template variables: `{{item.field_name}}` — no spaces inside braces, no array indexing

## Converting Between Formats

- **SFT → RFT**: Strip assistant messages (RFT last message must be `user`), add grader reference fields. Use `scripts/convert_dataset.py --format rft`.
- **SFT → DPO**: Generate rejected responses (run base model on same prompts, intentionally degrade good outputs, or use human ranking).
- **DPO → SFT**: Extract chosen responses from the preferred output.

# Deployment Formats

## Model Format and SKU Mapping

| Base model family | `model.format` | `sku.name` | Endpoint type |
|-------------------|---------------|------------|---------------|
| gpt-4.1-mini | `"OpenAI"` | `"Standard"` | Project |
| gpt-4.1-nano | `"OpenAI"` | `"Standard"` | Project |
| o4-mini (RFT) | `"OpenAI"` | `"Standard"` | Project |
| gpt-oss-20b | `"Microsoft"` | `"GlobalStandard"` | Cognitive Services |
| Ministral-3B | `"Mistral AI"` | `"GlobalStandard"` | Cognitive Services |
| Llama-3.3-70B | `"Meta"` | `"GlobalStandard"` | Cognitive Services |
| Qwen-3-32B | `"Alibaba"` | `"GlobalStandard"` | Cognitive Services |

**Format strings are case-sensitive.** `"Mistral AI"` works; `"mistral"` does not.

## Two Endpoint Types

**Project Endpoint** (OpenAI models): `https://<resource>.services.ai.azure.com/api/projects/<project>/openai/v1/`
- Use `openai.OpenAI(base_url=..., api_key=...)` — NOT `AzureOpenAI`

**Cognitive Services Endpoint** (OSS models): `https://<resource>.cognitiveservices.azure.com/openai/deployments/<name>/chat/completions?api-version=2025-04-01-preview`
- Use `openai.AzureOpenAI(azure_endpoint=..., api_key=..., api_version=...)`

## CLI Deployment (`az cognitiveservices`)

The CLI uses **different** format strings than the ARM REST API for OSS models:

```bash
az cognitiveservices account deployment create \
  --name <resource> \
  --resource-group <rg> \
  --deployment-name <name> \
  --model-name <model> \
  --model-version "1" \
  --model-format "OpenAI-OSS" \
  --sku-capacity 100 \
  --sku-name "GlobalStandard"
```

| Base model family | ARM REST `model.format` | CLI `--model-format` |
|-------------------|------------------------|----------------------|
| gpt-4.1-mini/nano | `"OpenAI"` | `"OpenAI"` |
| gpt-oss-20b | `"Microsoft"` | `"OpenAI-OSS"` |
| Ministral-3B | `"Mistral AI"` | `"OpenAI-OSS"` |
| Llama-3.3-70B | `"Meta"` | `"OpenAI-OSS"` |
| Qwen-3-32B | `"Alibaba"` | `"OpenAI-OSS"` |

> ⚠️ Using `"OpenAI-OSS"` in ARM REST or `"Microsoft"` in CLI will fail with HTTP 500.

## ARM REST API Deployment

```
PUT https://management.azure.com/subscriptions/{sub_id}/resourceGroups/{rg}/providers/Microsoft.CognitiveServices/accounts/{account}/deployments/{deploy_name}?api-version=2024-10-01
```

```json
{
  "sku": { "name": "GlobalStandard", "capacity": 100 },
  "properties": {
    "model": {
      "format": "Microsoft",
      "name": "gpt-oss-20b.ft-{jobid}-suffix",
      "version": "1"
    }
  }
}
```

**ARM token:** `az account get-access-token --query accessToken -o tsv` (expires ~60min).

## Capacity Notes

- Capacity = tokens-per-minute in thousands. `100` = 100K TPM.
- Set capacity ≥ 100 for eval workloads. At capacity=1, OSS FT models hit "Failed to load LoRA" errors.
- Quota is per-resource. After deleting a deployment, wait 15–20s before creating a new one.
- Deployment names: max 64 chars, alphanumeric + hyphens, unique within resource.

## Common Deployment Errors

| Error | Cause | Fix |
|-------|-------|-----|
| HTTP 500, no message | Wrong `model.format` | Check format table above |
| HTTP 409, deployment exists | Name collision | Use unique deployment name |
| HTTP 403 | ARM token expired | Refresh token |
| HTTP 400, "api-version not allowed" | `AzureOpenAI` client on `/v1/` endpoint | Switch to `openai.OpenAI` |
| HTTP 429, quota exceeded | Too many deployments | Delete unused, wait 20s |
| ProvisioningState: Failed | Model not available in region | Try different region |

# Evaluation Methodology

## Principles

1. **Always establish a baseline**: Evaluate the base (un-tuned) model first. Without a baseline, you can't measure improvement.
2. **Use a held-out test set**: Never evaluate on training or validation data. The model has seen those.
3. **Use the same test set for every model**: This is the only way to compare results fairly.
4. **Use task-specific graders**: Built-in generic evaluators (Coherence, Fluency) measure general quality and won't detect fine-tuning improvements. Use custom graders (Python, score model, string check) for task-specific evaluation.
5. **Measure cost alongside accuracy**: Report completion tokens per response when comparing models or checkpoints. A model that achieves the same accuracy with fewer tokens is strictly better — cheaper inference and lower latency.

## Two-Layer Evaluation Strategy

Use the **Azure AI Evaluation SDK** (`azure-ai-evaluation`) for all evaluation.

| Layer | Purpose | Grader Type | When |
|-------|---------|-------------|------|
| **Task-specific** (primary) | Measure FT improvement | `AzureOpenAIScoreModelGrader`, `AzureOpenAIPythonGrader`, `AzureOpenAIStringCheckGrader` | Every eval |
| **General quality** (guardrail) | Verify model didn't degrade | `CoherenceEvaluator`, `FluencyEvaluator` | Spot-check only |

Generic built-in evaluators (Coherence, Fluency, TaskAdherence) are guardrails, not metrics — they often show no difference between base and fine-tuned models even when domain-specific evaluation reveals clear improvement.

## Custom Graders (Primary FT Evaluation)

### 1. Score Model Grader (LLM judge with task-specific rubric)

Best for: subjective tasks (summarization, alignment, style).

```python
from azure.ai.evaluation import AzureOpenAIScoreModelGrader

summarization_grader = AzureOpenAIScoreModelGrader(
    model_config=model_config,
    name="summarization_quality",
    prompt="""Rate this news summary on a scale of 1-5.

Article: {{item.article}}
Summary: {{sample.output_text}}

Criteria:
- Captures ALL key facts (who, what, when, where)
- No hallucinated information not in the article
- Concise (under 3 sentences)

Score 1: Missing key facts or hallucinations
Score 3: Captures main point but misses details
Score 5: Perfect summary — all facts, no extras, concise

Return ONLY a number 1-5.""",
    output_type="numeric",
    pass_threshold=3,
)
```

### 2. Python Grader (programmatic/exact-match evaluation)

Best for: code generation, math, entity extraction, structured output.

```python
from azure.ai.evaluation import AzureOpenAIPythonGrader

entity_grader = AzureOpenAIPythonGrader(
    name="entity_extraction_accuracy",
    source="""
import json

def grade(item, sample):
    try:
        extracted = json.loads(sample["output_text"])
        reference = json.loads(item["ground_truth"])
    except (json.JSONDecodeError, KeyError):
        return {"score": 0, "reason": "Invalid JSON output"}

    required_keys = ["people", "organizations", "locations", "dates"]
    missing = [k for k in required_keys if k not in extracted]
    if missing:
        return {"score": 0.5, "reason": f"Missing keys: {missing}"}

    total, matched = 0, 0
    for key in required_keys:
        ref_set = set(str(v).lower() for v in reference.get(key, []))
        ext_set = set(str(v).lower() for v in extracted.get(key, []))
        total += len(ref_set)
        matched += len(ref_set & ext_set)

    score = matched / total if total > 0 else 1.0
    return {"score": score, "reason": f"{matched}/{total} entities matched"}
""",
    pass_threshold=0.7,
)
```

### 3. String Check Grader (pattern matching)

Best for: classification, format compliance, tool calling format.

```python
from azure.ai.evaluation import AzureOpenAIStringCheckGrader

tool_format_grader = AzureOpenAIStringCheckGrader(
    name="tool_call_format",
    input="{{sample.output_text}}",
    operation="like",          # or "eq", "starts_with", "contains"
    reference="function_call",
    pass_threshold=1,
)

classification_grader = AzureOpenAIStringCheckGrader(
    name="classification_accuracy",
    input="{{sample.output_text}}",
    operation="eq",
    reference="{{item.expected_label}}",
    pass_threshold=1,
)
```

## Running an Evaluation

The `evaluate()` function runs multiple graders over an entire dataset:

```python
from azure.ai.evaluation import evaluate, F1ScoreEvaluator

result = evaluate(
    data="eval_data.jsonl",
    evaluators={
        "task_grader": my_custom_score_grader,   # primary
        "f1": F1ScoreEvaluator(),                 # token overlap
    },
    output_path="./eval_results.json",
)

for metric, value in result["metrics"].items():
    print(f"{metric}: {value}")
```

## Test Set Design

- **Size**: 30–100 examples is sufficient.
- **Diversity**: Cover easy/medium/hard, edge cases, and different sub-categories.
- **Quality**: Reference answers must be gold-standard correct. A wrong reference penalizes correct outputs.

## Interpreting Results

| Score Type | Range | Meaning |
|-----------|-------|---------|
| AI quality (1–5) | 1–2 Poor, 3 Adequate, 4 Good, 5 Excellent | |
| NLP (0–1) | <0.3 Wrong, 0.3–0.6 Partial, 0.6–0.8 Good, >0.8 Strong | |

With 50+ eval examples, a difference of ~0.3 points (on 1–5 scale) is usually meaningful.

## Evaluating RFT Models

1. **Evaluate with a DIFFERENT rubric than the training grader** — otherwise you measure overfitting to the grader.
2. Use `F1ScoreEvaluator` for exact-match accuracy.
3. Use `SimilarityEvaluator` to catch semantically correct but differently formatted answers.
4. **Compare against the base model**, not just other fine-tunes.

## Reference

- [Azure AI Evaluation SDK docs](https://learn.microsoft.com/en-us/python/api/overview/azure/ai-evaluation-readme)
- [Evaluation samples](https://github.com/Azure-Samples/azureai-samples/tree/main/scenarios/evaluate)

# RFT Grader Design Guide

## Grader Type Selection

| Grader Type | Best For | Tradeoffs |
|------------|---------|-----------|
| **Python grader** (default) | Most tasks incl. tool-calling. Accesses `output_text` and `output_tools`. | Can't call external APIs or execute code. |
| **Multi grader** | Combining multiple scoring dimensions. | `score_model` component adds LLM cost per rollout. |
| **Endpoint grader** | Tasks requiring external API calls (test suites, DB queries). | HTTP latency, scaling risk. Under-provisioned endpoints can hang jobs. |
| **String check** | Exact-match tasks (classification, yes/no, numeric). | Binary 0/1 only — no partial credit. |

Start with Python grader unless you need external API calls. Python graders are fast, deterministic, reliable, and tool-aware (`sample.output_tools` provides tool call metadata).

## Partial Credit Pattern

Binary pass/fail gives sparse reward. Decompose into 2–4 scored dimensions:

```python
def grade(sample, item):
    output_text = sample.get("output_text", "") or ""
    expected = item.get("expected_answer", "")
    
    score = 0.0
    
    # Core correctness (highest weight)
    if correct_action(output_text, expected):
        score += 0.4
    
    # Precision (exact amounts, specific values)
    score += 0.3 * precision_score(output_text, expected)
    
    # Reasoning quality (cited correct rules/facts)
    score += 0.2 * reasoning_score(output_text, expected)
    
    # Process quality (used the right tools)
    if used_correct_tools(sample.get("output_tools", [])):
        score += 0.1
    
    return round(min(score, 1.0), 3)
```

### Weight Guidelines

| Dimension | Typical Weight | Examples |
|-----------|---------------|----------|
| Core correctness | 0.3–0.5 | Right action/answer/classification |
| Precision | 0.2–0.3 | Exact amounts, correct format |
| Reasoning | 0.1–0.2 | Cited correct rules, justified decision |
| Process quality | 0.05–0.1 | Used right tools, followed steps |

## Threshold Calibration Workflow

The `pass_threshold` determines what score counts as pass vs fail — the most important RFT hyperparameter.

1. Run the **base model** on your training/validation set
2. Score every output with your grader
3. Compute pass rates at multiple thresholds:

```python
for threshold in [0.5, 0.6, 0.7, 0.8, 0.85, 0.9, 0.95]:
    pass_rate = sum(1 for s in scores if s >= threshold) / len(scores)
    print(f"  @{threshold}: pass={pass_rate:.0%}, fail={1 - pass_rate:.0%}")
```

4. Choose where **25–50% of base model rollouts fail**:

| Failure Rate | Signal Quality |
|-------------|----------------|
| < 10% | ❌ Too easy — no learning signal |
| 10–25% | ⚠️ Weak signal |
| **25–50%** | ✅ Good — enough failures to learn from |
| 50–70% | ⚠️ Harsh — mostly negative reward |
| > 70% | ❌ Too hard — training may diverge |

**Always re-run calibration when you change your dataset.**

## Consistency Rules

When using multiple graders (Python for training, endpoint for debugging, local script for eval):

1. **Identical scoring logic** — same weights, keywords, dimension breakdown
2. **Identical default scores** — same behavior when no action found, no amounts expected
3. **Test with same examples** — run 10 samples through all graders and verify scores match

Mismatched scoring causes the model to learn different behavior than what your evaluation measures.

# Hyperparameter Guide

## SFT / DPO Core Parameters

| Parameter | What it controls | Default | Typical range |
|-----------|-----------------|---------|---------------|
| **Epochs** | Passes through data | 2 | 1–5 |
| **Learning rate multiplier** | Weight change aggressiveness | 1.0 | 0.1–2.0 |
| **Batch size** | Examples per gradient step | Model-dependent | 4–32 |

### Dataset Size vs Epochs

| Dataset size | Recommended epochs |
|-------------|-------------------|
| < 100 examples | 3–5 |
| 100–500 examples | 2–3 |
| 500–2,000 examples | 1–2 |
| > 2,000 examples | 1 |

### Learning Rate Guidelines
- **Higher LR** (1.5–2.0): Large/diverse datasets, task very different from pre-training
- **Lower LR** (0.1–0.5): Small datasets (<200), refining not overwriting base behavior
- For 1,000+ examples, LR 0.2–0.5 often beats default 1.0

### DPO-Specific Parameters
- `beta` (default 0.1): Alignment strength. Lower = more conservative.
- `l2_multiplier` (default 0.1): Regularization to prevent drift from base model.

## HP Sweep Strategy

| Run | Epochs | LR | Why |
|-----|--------|----|-----|
| 1 | 2 | 1.0 | Baseline |
| 2 | 2 | 0.5 | Conservative |
| 3 | 2 | 1.5 | Aggressive |
| 4 | 3 | 1.0 | More training |
| 5 | 1 | 1.0 | Minimal intervention |

## Checkpoint Trick

When overfitting (val loss rises after epoch 2): deploy the epoch-2 checkpoint directly instead of retraining. Azure saves checkpoints at each epoch boundary.

```python
checkpoints = client.fine_tuning.jobs.checkpoints.list(job_id)
for cp in checkpoints.data:
    print(f"Step {cp.step_number}: val_loss={cp.metrics.valid_loss}")
```

## Model-Specific Recommendations

| Model | Recommended Start | Notes |
|-------|------------------|-------|
| gpt-4.1-mini | 2ep, lr=0.5–1.0 | Very capable base; small nudges work |
| gpt-4.1-nano | 2–3ep, lr=1.0–1.5 | Smaller capacity, needs more epochs |
| gpt-oss-20b | 2ep, lr=0.2–0.5 | Lower LR critical; deployment may need capacity=100 |
| o4-mini (RFT) | Grader quality > HPs | Focus on grader, not HP sweep |

## OSS Model Parameters

All OSS models require `trainingType: "globalStandard"` in the API request.

| Model | Recommended Start | Best Found | Notes |
|-------|------------------|------------|-------|
| Ministral-3B | 5ep, lr=1.0 | 10ep, lr=0.5 | Small model, slow convergence |
| gpt-oss-20b | 2ep, lr=0.3 | 2ep, lr=0.3 | lr=1.0 overfits quickly |
| Llama-3.3-70B | 3ep, lr=0.3 | 5ep, lr=0.5 | lr=2.0 causes catastrophic degradation |
| Qwen-3-32B | 3ep, lr=0.3 | 3ep, lr=0.3 | Most fragile — more data can hurt |

**Key patterns**: OSS models need 2–5× more epochs than nano. Lower LR (0.3–0.5) is safer. More data doesn't always help.

## RFT Hyperparameters

| Parameter | Description | Recommended Start |
|-----------|-------------|-------------------|
| `reasoning_effort` | `"low"`, `"medium"`, `"high"` | `"medium"` |
| `compute_multiplier` | Scales rollouts per step | `1.5` |
| `learning_rate_multiplier` | Scales LR | `1.0` |
| `n_epochs` | Data passes | `2–3` |
| `eval_interval` | Eval every N steps | `5` |
| `eval_samples` | Validation examples per eval | `10` |
| `max_episode_steps` | Max tool calls + reasoning steps | `5–10` |

# Large File Uploads

The standard `client.files.create()` silently fails on JSONL files >~150MB (Azure returns 500 during job execution). Use the chunked Uploads API:

```python
upload = client.uploads.create(filename="data.jsonl", purpose="fine-tune", bytes=file_size, mime_type="application/jsonl")
part_ids = []
with open(filepath, "rb") as f:
    while chunk := f.read(64 * 1024 * 1024):  # 64MB chunks
        part = client.uploads.parts.create(upload_id=upload.id, data=chunk)
        part_ids.append(part.id)
completed = client.uploads.complete(upload_id=upload.id, part_ids=part_ids)
file_id = completed.file.id
```

**Important:** Requires `openai.AzureOpenAI()` client, NOT `openai.OpenAI()` with `/v1/` URL. The project endpoint returns 404 for upload operations.

| File Size | Method |
|-----------|--------|
| < 100MB | Standard `files.create()` |
| 100MB–5GB | Chunked Uploads API |
| > 5GB | Split dataset |

# Platform Gotchas — Top 10

1. **OSS models require `"trainingType": "globalStandard"`** in the request body — undocumented, and all OSS FT jobs fail without it.

2. **Model catalog `fine_tune` flag is wrong for OSS models** — API returns `fine_tune = false` for all OSS models despite being FT-supported. Hardcode the supported list.

3. **Older SDK versions may fail on `/v1/` project endpoints** — `client.files.create()` throws "API version not supported" with older `openai` package versions. Upgrade to `openai>=1.0` and use the `/v1/` project endpoint (preferred). If you must use an older SDK, fall back to REST API with the non-project `/openai/` endpoint.

4. **ARM "Succeeded" doesn't mean deployment is ready** — `provisioningState: Succeeded` but data plane returns `DeploymentNotReady` indefinitely. Delete and recreate the deployment, then wait ~5 minutes.

5. **OSS FT deployments may fail with InternalServerError** — use the correct provider-specific `model.format` (e.g., `"Mistral AI"` not `"OpenAI"`) and try `capacity=100`.

6. **OSS FT inference hits "Failed to load LoRA" intermittently** — deploy with capacity ≥ 100, use 8+ retries with exponential backoff, and wait 2+ minutes after deployment before first call.

7. **ARM REST and `az cognitiveservices` use different format strings for OSS models** — ARM uses provider names (`"Microsoft"`, `"Meta"`), CLI uses `"OpenAI-OSS"` for all OSS. Mixing them produces HTTP 500.

8. **Content safety false positives on entity extraction data** — PII-dense data (medical records, legal docs, resumes) can trigger "Hate/Fairness" blocks at deployment time. Remove problematic document types.

9. **FT deployments at capacity=1 are severely rate-limited (~1 RPM)** — evaluating 10 samples takes ~10 minutes. Use capacity ≥ 100 for eval workloads and exponential backoff.

10. **Wrong resource endpoint is a silent killer** — jobs submitted to the wrong Foundry resource succeed via API but don't appear in the portal. Always verify the endpoint matches your Foundry project.

# Reward Hacking Prevention in RFT

## What Is Reward Hacking?

The model optimizes for the grader's scoring function rather than the actual task. The training grader becomes a proxy reward that diverges from true quality — the model games the proxy instead of improving.

**Core rule: Your training grader MUST produce the same ranking as your evaluation methodology.**

| If you evaluate with… | Then train with… | NOT with… |
|------------------------|------------------|-----------|
| LLM judge (semantic) | LLM judge | AST / regex / structural matching |
| Exact match | Exact match | Fuzzy or partial matching |
| Unit tests | Unit tests | Static analysis alone |

Misaligned graders are the #1 cause of reward hacking.

## Train-Val Gap Thresholds

| Train-Val Gap | Status | Action |
|---------------|--------|--------|
| ≤ 0.05 | ✅ Healthy | Continue training |
| 0.05–0.10 | ⚠️ Warning | Monitor closely, check outputs qualitatively |
| > 0.10 | 🛑 Stop | Stop training — reward hacking is likely |

## Pre-Training Checklist

1. **Baseline the grader**: Run training grader on base model outputs. Record scores as your floor.
2. **Cross-validate graders**: If training grader ≠ eval grader, generate 50 outputs, score with both, compute Spearman ρ. Proceed only if ρ ≥ 0.8. If ρ < 0.6, fix alignment first.
3. **Test hackability**: Generate 5 intentionally bad outputs that might score well. If grader scores any > 5/10, redesign it.
4. **Set gap threshold**: Monitor train-val gap every eval_interval. Stop if > 0.10.

## Grader Iteration Loop

When reward hacking is detected:

```
1. STOP the training run
        ↓
2. COLLECT "hacked" outputs (high train score, low eval score)
        ↓
3. ANALYZE what pattern the model exploited
   (structural mimicry? verbosity? keyword stuffing?)
        ↓
4. UPDATE the grader to penalize that pattern
        ↓
5. RE-BASELINE the updated grader on base model outputs
        ↓
6. RESTART training with the improved grader
```

## Red Flags Checklist

Investigate immediately if **any** are true:

- [ ] Train-val gap > 0.10
- [ ] Training reward increasing but eval quality stable or declining
- [ ] Model outputs are longer/more verbose than base model
- [ ] Outputs structurally match references but are semantically wrong
- [ ] Different LLM judges disagree on quality
- [ ] Conciseness/style scores dropping while correctness climbs
- [ ] Model produces "template" responses

## Key Principles

| Principle | Action |
|-----------|--------|
| Align graders | Training grader must rank outputs same as eval |
| Cross-validate first | Spearman ρ ≥ 0.8 between training and eval graders |
| Monitor train-val gap | ≤ 0.05 healthy, > 0.10 stop |
| Test hackability | Bad outputs should score < 5/10 |
| Prefer SFT when possible | Use RFT only for verifiable-answer tasks |
| Iterate graders, not models | Fix grader before restarting training |

# Training Curve Analysis

## SFT Metrics

| Column | What it means |
|--------|---------------|
| `train_loss` | Loss on training batch (should decrease) |
| `train_mean_token_accuracy` | Token-level accuracy on training data |
| `valid_loss` | Loss on validation set (**primary metric**) |
| `valid_mean_token_accuracy` | Token-level accuracy on validation data |
| `full_valid_loss` | Full-pass validation loss (more accurate, less frequent) |
| `full_valid_mean_token_accuracy` | Full-pass token accuracy |

## Overfitting Detection

**Overfitting ratio** at each checkpoint: `valid_loss / train_loss`

| Ratio | Interpretation |
|-------|---------------|
| < 1.2 | Healthy — generalizes well |
| 1.2–1.5 | Mild overfitting — acceptable for small datasets |
| 1.5–2.0 | Moderate — consider reducing epochs |
| > 2.0 | Severe — deploy an earlier checkpoint |

```python
val_losses = [cp.metrics.valid_loss for cp in checkpoints if cp.metrics.valid_loss]
best_val = min(val_losses)
final_val = val_losses[-1]
if final_val > best_val * 1.2:
    print(f"⚠️ OVERFIT: Best={best_val:.4f}, final={final_val:.4f}")
```

## Best Checkpoint Selection (SFT)

```python
checkpoints = client.fine_tuning.jobs.checkpoints.list(job_id)
best_cp = min(checkpoints.data, key=lambda cp: cp.metrics.valid_loss or float('inf'))
print(f"Best: step {best_cp.step_number}, valid_loss={best_cp.metrics.valid_loss:.4f}, "
      f"model={best_cp.fine_tuned_model_checkpoint}")
```

## Diagnosis Table

| Observation | Diagnosis | Action |
|-------------|-----------|--------|
| Train loss barely decreases | LR too low or noisy data | Increase LR or clean data |
| Train loss crashes to ~0 | LR too high or easy data | Decrease LR or add harder examples |
| Valid loss rises after epoch 2 | Overfitting | Deploy epoch-2 checkpoint |
| Valid loss plateaus after epoch 1 | Learned quickly | Try epoch=1 or lower LR |
| Valid loss oscillates | Small batch or inconsistent data | Increase batch size or audit data |
| Both losses stay high | Task too hard | Larger model or simplify task |
| Large train-valid gap from start | Insufficient/mismatched data | Add diverse training data |

## RFT Metrics

| Column | What it means |
|--------|---------------|
| `train_mean_reward` | Average reward across rollouts (**primary** — should increase) |
| `full_valid_mean_reward` | Validation reward (overfitting check) |
| `completion_tokens_mean` | Average response length per rollout |
| `reasoning_tokens_mean` | Average reasoning tokens (o-series models) |
| `mean_unresponsive_rewards` | Rollouts with no scoreable output |
| `train_sample_parse_error_count` | Grader couldn't parse output |
| `train_other_error_count` | Grader logic bugs — should be 0 |

## RFT Reward Curve Patterns

- **Reward flat at ~0**: Grader broken or threshold too strict
- **Reward always negative**: pass_threshold too high
- **Reward immediately high + flat**: Threshold too lenient
- **Train-valid reward gap > 0.10**: Possible reward hacking

### Token Growth
- **Moderate** (tokens double): Normal — model becoming more thorough
- **Excessive** (3x+): Grader may incentivize verbosity — check scoring dimensions
- When comparing checkpoints, equal accuracy at fewer tokens is strictly better

### Parse Errors vs Logic Errors
- `sample_parse_error_count`: Often high in agentic RFT (mid-reasoning captures). Training still works if reward is climbing.
- `other_error_count`: Bugs in grader logic. Fix before continuing.

## RFT Checkpoint Selection

```python
checkpoints = client.fine_tuning.jobs.checkpoints.list(job_id)
for cp in checkpoints:
    m = cp.metrics
    tr = f"{m.train_mean_reward:.3f}" if m.train_mean_reward is not None else "n/a"
    vr = f"{m.full_valid_mean_reward:.3f}" if m.full_valid_mean_reward is not None else "n/a"
    ct = f"{m.completion_tokens_mean:.0f}" if m.completion_tokens_mean is not None else "n/a"
    print(f"Step {cp.step_number}: train_reward={tr}, valid_reward={vr}, tokens={ct}")
```

Don't rely solely on `valid_reward` for RFT — deploy 2–3 candidates (peak reward, final, mid-training) and evaluate with your real task harness including tool execution.

# Training Types: SFT vs DPO vs RFT

## Decision Matrix

| Factor | SFT | DPO | RFT |
|--------|-----|-----|-----|
| **Best for** | Teaching a new skill or format | Aligning preferences/style | Improving reasoning chains |
| **Data needed** | Input–output pairs | Chosen/rejected pairs | Prompts + grading function |
| **Data volume** | 50–5,000 examples | 500–5,000 pairs | 200–2,000 prompts |
| **Effort to prepare data** | Low | High (need contrasting pairs) | Medium (need grader, not outputs) |
| **Risk of regression** | Low | Medium | High (sensitive to grader quality) |
| **Typical improvement** | 5–30% on task metrics | Subtle style/safety shifts | 0–15% on reasoning tasks |
| **Supported models** | Most models | Select models | o4-mini |

## When to Use Each

### SFT (Supervised Fine-Tuning)
- You have high-quality input–output pairs
- Task is well-defined (code generation, classification, extraction, summarization)
- You want reliable, repeatable outputs in a specific format or style
- **Key insight**: 300–500 high-quality examples often outperforms 1,500+ lower-quality ones

### DPO (Direct Preference Optimization)
- You want to adjust tone, verbosity, safety, or style
- You have examples of "good" and "bad" outputs for the same input
- SFT already works but outputs need refinement
- DPO-specific params: `beta` (default 0.1), `l2_multiplier` (default 0.1)

### RFT (Reinforcement Fine-Tuning)
- Task has objectively verifiable answers (code execution, math, logic)
- You can write a programmatic or LLM-based grader
- You want to improve the model's reasoning, not just its outputs
- **Critical**: RFT is extremely sensitive to grader quality. Train–val gap should be ≤ 0.05.

## Choosing a Path

```
├─ Do you have labeled input–output pairs?
│  ├─ Yes → SFT
│  └─ No
│     ├─ Can you write a grading function? → RFT
│     └─ Can you rank "good" vs "bad" outputs? → DPO
│
After SFT:
├─ Results good enough? → Ship it
├─ Need style refinement? → DPO on top of SFT model
└─ Reasoning needs improvement? → RFT (if model supports it)
```

## Model Compatibility (Azure AI Foundry)

| Model | SFT | DPO | RFT | Vision FT |
|-------|-----|-----|-----|-----------|
| gpt-4.1 | ✅ | ✅ | ❌ | ✅ |
| gpt-4.1-mini | ✅ | ❌ | ❌ | ❌ |
| gpt-4.1-nano | ✅ | ❌ | ❌ | ❌ |
| gpt-4o (2024-08-06) | ✅ | ✅ | ❌ | ✅ |
| gpt-4o-mini | ✅ | ❌ | ❌ | ❌ |
| o4-mini | ❌ | ❌ | ✅ | ❌ |
| gpt-5 | ❌ | ❌ | ✅ ⚠️ | ❌ |
| gpt-oss-20b | ✅ | ❌ | ❌ | ❌ |
| Ministral-3B | ✅ | ❌ | ❌ | ❌ |
| Llama-3.3-70B | ✅ | ❌ | ❌ | ❌ |
| Qwen-3-32B | ✅ | ❌ | ❌ | ❌ |

DPO can be applied on top of an already SFT-fine-tuned model. Vision fine-tuning follows the same SFT workflow but with image data in messages.

> ⚠️ **Feature flags**: GPT-5 RFT and agentic RFT with tool calling require access requests. Contact your Microsoft account team or request access through the Azure AI Foundry portal. o4-mini RFT without tools is generally available.

*Check Azure AI Foundry docs for the latest model availability.*

# Vision Fine-Tuning

Fine-tune models with image data to customize visual understanding. Uses the same chat-completions JSONL format as text SFT, but with image content blocks in user messages.

## Supported Models

| Model | Version |
|-------|---------|
| gpt-4o | 2024-08-06 |
| gpt-4.1 | 2025-04-14 |

## Image Requirements

| Constraint | Limit |
|-----------|-------|
| Max examples with images per training file | 50,000 |
| Max images per example | 64 |
| Max image file size | 10 MB |
| Supported formats | JPEG, PNG, WEBP |
| Color mode | RGB or RGBA |
| Min examples | 10 |

**Important**: Images can only appear in `user` messages, never in `assistant` responses.

## Data Format

Each training example follows the standard SFT `messages` format. Images are included as `image_url` content blocks within user messages.

```jsonl
{"messages": [{"role": "system", "content": "You are a helpful AI assistant that describes images."}, {"role": "user", "content": [{"type": "text", "text": "Describe this image."}, {"type": "image_url", "image_url": {"url": "https://example.com/photo.png", "detail": "high"}}]}, {"role": "assistant", "content": "The image shows a cityscape with tall buildings against a blue sky."}]}
```

### Image Sources

Images can be provided in two ways:

**1. Public URL:**
```json
{"type": "image_url", "image_url": {"url": "https://example.com/image.png"}}
```

**2. Base64 data URI:**
```json
{"type": "image_url", "image_url": {"url": "data:image/png;base64,iVBORw0KGgo..."}}
```

### Detail Control

The `detail` parameter controls image processing fidelity and cost:

| Value | Behavior | Cost |
|-------|----------|------|
| `low` | Downscales to 512×512 pixels | Lower |
| `high` | Full resolution processing | Higher |
| `auto` | Model decides based on image size | Default |

```json
{"type": "image_url", "image_url": {"url": "https://example.com/image.png", "detail": "low"}}
```

Use `low` for tasks where fine visual detail doesn't matter (classification, general description). Use `high` for tasks needing precise detail (OCR, diagram reading, defect detection).

## Content Moderation

Images are screened before training. The following are **automatically excluded**:

- Images containing **people or faces** (face detection only — no identification)
- **CAPTCHAs**
- Content violating Azure usage policies

This screening may add latency to file upload validation.

## Best Practices

- **Diverse examples**: Vary image content, angles, lighting, and resolution
- **Consistent annotations**: Keep assistant response style and detail level uniform
- **Start with `detail: low`**: Cheaper and faster — upgrade to `high` only if results need it
- **Check for excluded images**: After upload, verify the training count matches expectations — some images may be silently skipped due to content moderation
- **Mixed text+image**: You can include both text-only and image examples in the same training file

## Training Workflow

Vision fine-tuning follows the exact same workflow as text SFT:

1. Prepare JSONL with image content blocks
2. Upload training file (validation may take longer due to image screening)
3. Create fine-tuning job with a supported vision model
4. Monitor and evaluate as usual

```python
# Upload (image validation may take longer)
train_file = client.files.create(purpose="fine-tune", file=open("vision_train.jsonl", "rb"))
client.files.wait_for_processing(train_file.id)

# Submit — same as text SFT
job = client.fine_tuning.jobs.create(
    model="gpt-4.1-2025-04-14",
    training_file=train_file.id,
    validation_file=val_file.id,
    method={"type": "supervised"}
)
```

## Troubleshooting

| Issue | Resolution |
|-------|-----------|
| Images skipped silently | Check for people/faces, oversized files, unsupported formats |
| URL not accessible | Ensure URLs are publicly accessible, or use base64 data URIs |
| Exceeds 10 MB | Resize or compress the image |
| Wrong color mode | Convert to RGB or RGBA |
| Low quality results | Try `detail: high`, add more diverse examples, increase dataset size |

## Reference

- [Official docs](https://learn.microsoft.com/en-us/azure/foundry/openai/how-to/fine-tuning-vision)

# /// script
# dependencies = [
#   "openai>=1.0",
#   "azure-identity",
#   "azure-ai-projects",
# ]
# ///
"""
calibrate_grader.py — Calibrate RFT grader pass_threshold before submitting a job.

Runs the base model on your training/validation data, scores each output
with your Python grader, and recommends the optimal pass_threshold.

Usage:
  python calibrate_grader.py --base-url <url> --api-key KEY \
      --model o4-mini --data train.jsonl --grader grader.py --n 30

  python calibrate_grader.py --model gpt-4.1-mini --data val.jsonl \
      --grader grader.py --n 20 --tools '[{"name": "search", "server_url": "https://..."}]'
"""

import argparse
import json
import os
import random
import sys

try:
    sys.stdout.reconfigure(encoding="utf-8")
    sys.stderr.reconfigure(encoding="utf-8")
except (AttributeError, OSError):
    pass  # Stream not reconfigurable (older Python or non-tty); default encoding is fine
import time

sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from common import HelpOnErrorParser, get_clients


def load_grader(grader_path):
    """Load and compile a Python grader file. Returns the grade() function.

    SECURITY: This executes the grader file as Python code. Only load grader
    files that you wrote or reviewed — never load untrusted files from the
    internet or unknown sources. The grader runs with the same permissions as
    this script.
    """
    grader_path = os.path.abspath(grader_path)
    if not os.path.isfile(grader_path):
        print(f"❌ Grader file not found: {grader_path}")
        sys.exit(1)
    with open(grader_path, encoding="utf-8") as f:
        source = f.read()
    namespace = {}
    exec(compile(source, grader_path, "exec"), namespace)
    if "grade" not in namespace:
        print(f"❌ Grader file must define a grade(sample, item) function")
        sys.exit(1)
    return namespace["grade"]


def run_model(client, model, messages, tools_schema=None, max_retries=3):
    """Run the model and return (output_text, output_tools)."""
    kwargs = {"model": model, "messages": messages, "max_completion_tokens": 4096}
    if tools_schema:
        kwargs["tools"] = tools_schema

    for attempt in range(max_retries):
        try:
            resp = client.chat.completions.create(**kwargs)
            msg = resp.choices[0].message
            output_text = msg.content or ""
            output_tools = []
            if msg.tool_calls:
                output_tools = [
                    {"type": "function", "function": {"name": tc.function.name, "arguments": tc.function.arguments}}
                    for tc in msg.tool_calls
                ]
            return output_text, output_tools
        except Exception as e:
            if "429" in str(e) and attempt < max_retries - 1:
                time.sleep(5 * (attempt + 1))
            else:
                return f"ERROR: {e}", []
    return "ERROR: max retries", []


def calibrate(client, model, data, grade_fn, tools_schema=None, n=30):
    """Run base model on data, score with grader, output threshold analysis."""
    if not data:
        print("No examples to evaluate. Check your data file.")
        return

    # Sample if dataset is larger than n
    if len(data) > n:
        data = random.sample(data, n)

    print(f"Running {model} on {len(data)} examples...\n")

    scores = []
    for i, ex in enumerate(data):
        messages = ex["messages"]
        user_msg = messages[-1]["content"] if messages else ""

        output_text, output_tools = run_model(client, model, messages, tools_schema)

        if output_text.startswith("ERROR:"):
            print(f"  [{i+1:3d}] ❌ {output_text[:60]}")
            scores.append(0.0)
            continue

        # Build sample dict matching what the grader expects
        sample = {"output_text": output_text, "output_tools": output_tools}

        # Build item dict from all fields in the training example
        item = {k: v for k, v in ex.items() if k != "messages"}

        try:
            score = grade_fn(sample, item)
        except Exception as e:
            print(f"  [{i+1:3d}] ❌ Grader error: {e}")
            scores.append(0.0)
            continue

        status = "✅" if score >= 0.9 else ("⚠️" if score >= 0.5 else "❌")
        print(f"  [{i+1:3d}] {score:.3f} {status}  {user_msg[:55]}")
        scores.append(score)

        time.sleep(0.5)  # Rate limiting

    # Analysis
    scored = [s for s in scores if s is not None]
    if not scored:
        print("\n❌ No examples were scored successfully. Check model access and data format.")
        return
    avg = sum(scored) / len(scored)
    print(f"\n{'='*60}")
    print(f"  BASE MODEL GRADER CALIBRATION ({len(scores)} examples)")
    print(f"  Average score: {avg:.1%}")
    print(f"{'='*60}")

    print(f"\n  {'Threshold':>10} {'Pass Rate':>10} {'Fail Rate':>10} {'Signal':>20}")
    print(f"  {'-'*10} {'-'*10} {'-'*10} {'-'*20}")

    best_threshold = None
    best_distance = float("inf")

    for threshold in [0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.85, 0.9, 0.95, 1.0]:
        pass_rate = sum(1 for s in scored if s >= threshold) / len(scored)
        fail_rate = 1 - pass_rate

        if 0.25 <= fail_rate <= 0.50:
            signal = "✅ Good (25-50%)"
            distance = abs(fail_rate - 0.35)  # Ideal is ~35%
            if distance < best_distance:
                best_distance = distance
                best_threshold = threshold
        elif fail_rate < 0.10:
            signal = "❌ Too easy"
        elif fail_rate < 0.25:
            signal = "⚠️ Weak signal"
        elif fail_rate <= 0.70:
            signal = "⚠️ Harsh"
        else:
            signal = "❌ Too hard"

        print(f"  {threshold:>10.2f} {pass_rate:>9.0%} {fail_rate:>9.0%} {signal:>20}")

    if best_threshold:
        print(f"\n  ✅ Recommended pass_threshold: {best_threshold}")
        print(f"     (~{sum(1 for s in scores if s < best_threshold)/len(scores):.0%} failure rate)")
    else:
        print(f"\n  ⚠️ No threshold in the ideal 25-50% failure range.")
        print(f"     Consider adjusting your grader scoring dimensions.")

    # Score distribution
    print(f"\n  Score distribution:")
    buckets = {"0.0-0.2": 0, "0.2-0.4": 0, "0.4-0.6": 0, "0.6-0.8": 0, "0.8-0.9": 0, "0.9-1.0": 0}
    for s in scores:
        if s < 0.2: buckets["0.0-0.2"] += 1
        elif s < 0.4: buckets["0.2-0.4"] += 1
        elif s < 0.6: buckets["0.4-0.6"] += 1
        elif s < 0.8: buckets["0.6-0.8"] += 1
        elif s < 0.9: buckets["0.8-0.9"] += 1
        else: buckets["0.9-1.0"] += 1
    for bucket, count in buckets.items():
        bar = "█" * count
        print(f"    {bucket}: {count:3d} {bar}")


def build_parser():
    parser = HelpOnErrorParser(
        description="Calibrate RFT grader pass_threshold on base model outputs",
        epilog=(
            "Example:\n"
            "  python calibrate_grader.py --model o4-mini --data train.jsonl --grader grader.py\n"
            "  python calibrate_grader.py --model o4-mini --data val.jsonl --grader grader.py --n 20"
        ),
        formatter_class=argparse.RawTextHelpFormatter,
    )
    parser.add_argument("--base-url", default=os.environ.get("OPENAI_BASE_URL"), help="Project /v1/ endpoint URL")
    parser.add_argument("--endpoint", default=os.environ.get("AZURE_OPENAI_ENDPOINT"),
                        help="Azure OpenAI endpoint (fallback)")
    parser.add_argument("--api-key", default=os.environ.get("AZURE_OPENAI_API_KEY"), help="API key")
    parser.add_argument("--project-endpoint", default=os.environ.get("AZURE_AI_PROJECT_ENDPOINT"),
                        help="Azure AI project endpoint")
    parser.add_argument("--model", required=True, help="Base model deployment name to calibrate against")
    parser.add_argument("--data", required=True, help="Path to training or validation JSONL file")
    parser.add_argument("--grader", required=True, help="Path to Python grader file (must define grade(sample, item))")
    parser.add_argument("--n", type=int, default=30, help="Number of examples to evaluate (default: 30)")
    parser.add_argument("--tools", default=None,
                        help="Tool schemas as JSON array (for tool-calling models). Pass as a JSON string.")
    parser.add_argument("--seed", type=int, default=42, help="Random seed for sampling (default: 42)")
    return parser


if __name__ == "__main__":
    parser = build_parser()
    if len(sys.argv) == 1:
        parser.print_help()
        sys.exit(0)

    args = parser.parse_args()
    random.seed(args.seed)

    client, method = get_clients(base_url=args.base_url, azure_endpoint=args.endpoint, project_endpoint=args.project_endpoint, api_key=args.api_key)

    # Load data
    with open(args.data, encoding="utf-8") as f:
        data = []
        for ln, line in enumerate(f, 1):
            if not line.strip():
                continue
            try:
                data.append(json.loads(line))
            except json.JSONDecodeError as e:
                print(f"⚠️ Skipping malformed JSON on line {ln}: {e}")
    print(f"Loaded {len(data)} examples from {args.data}")

    # Load grader
    grade_fn = load_grader(args.grader)
    print(f"Loaded grader from {args.grader}")

    # Parse tools if provided
    tools_schema = None
    if args.tools:
        tools_schema = json.loads(args.tools)

    calibrate(client, args.model, data, grade_fn, tools_schema, args.n)

# /// script
# dependencies = [
#   "openai>=1.0",
#   "azure-identity",
#   "azure-ai-projects",
# ]
# ///
"""
check_training.py — Analyze training curves, detect overfitting, list checkpoints.

Usage:
  python check_training.py --job-id ftjob-abc123
  python check_training.py --job-id ftjob-abc123 --download-csv results.csv
  python check_training.py --base-url https://<resource>.services.ai.azure.com/api/projects/<project>/openai/v1/ --api-key KEY --job-id ftjob-abc123
"""

import csv
import io
import os
import sys

try:
    sys.stdout.reconfigure(encoding="utf-8")
    sys.stderr.reconfigure(encoding="utf-8")
except (AttributeError, OSError):
    pass  # Stream not reconfigurable (older Python or non-tty); default encoding is fine
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from common import HelpOnErrorParser, get_clients


def analyze_job(client, job_id, download_csv=None):
    """Pull training results, analyze curves, detect overfitting."""
    job = client.fine_tuning.jobs.retrieve(job_id)

    print(f"Job: {job.id}")
    print(f"  Model: {job.model}")
    print(f"  Status: {job.status}")
    print(f"  Fine-tuned model: {job.fine_tuned_model}")

    if job.hyperparameters:
        hp = job.hyperparameters
        print(f"  Epochs: {getattr(hp, 'n_epochs', 'N/A')}")
        print(f"  LR multiplier: {getattr(hp, 'learning_rate_multiplier', 'N/A')}")
        print(f"  Batch size: {getattr(hp, 'batch_size', 'N/A')}")

    # Allow analysis while still running if result files exist
    if job.status not in ("succeeded", "running"):
        print(f"\n  Job status is '{job.status}'. Cannot analyze curves.")
        return

    if not job.result_files:
        if job.status == "running":
            print("\n  Job is still running and no result files available yet. Check back later.")
        else:
            print("\n  No result files available.")
        return

    # Download results CSV
    content = client.files.content(job.result_files[0])
    csv_data = content.read()

    if download_csv:
        with open(download_csv, "wb") as f:
            f.write(csv_data)
        print(f"\n  Results CSV saved to {download_csv}")

    # Parse CSV
    reader = csv.DictReader(io.StringIO(csv_data.decode("utf-8")))
    rows = list(reader)

    if job.status == "running":
        print(f"\n  ⚡ Job still running — showing partial results ({len(rows)} steps so far)")

    # Extract validation checkpoints
    val_points = []
    for row in rows:
        step = int(row.get("step", 0))
        train_loss = float(row["train_loss"]) if row.get("train_loss", "").strip() else None
        val_loss = None
        for col in ["valid_loss", "full_valid_loss", "eval_loss"]:
            if row.get(col, "").strip():
                val_loss = float(row[col])
                break

        if val_loss is not None:
            val_points.append((step, val_loss, train_loss))

    if not val_points:
        print("\n  No validation loss data found in results CSV.")
        return

    # Find best validation checkpoint
    best_step, best_val, best_train = min(val_points, key=lambda x: x[1])
    final_step, final_val, final_train = val_points[-1]

    print(f"\n  Training Curve Analysis:")
    print(f"  {'Step':>6} {'Val Loss':>10} {'Train Loss':>12} {'Ratio':>8}")
    print(f"  {'─'*6} {'─'*10} {'─'*12} {'─'*8}")
    for step, val, train in val_points:
        ratio = val / train if train and train > 0 else 0
        marker = " ← best" if step == best_step else ""
        train_str = f"{train:12.4f}" if train is not None else "         N/A"
        print(f"  {step:>6} {val:>10.4f} {train_str} {ratio:>8.2f}{marker}")

    print(f"\n  Best val_loss: {best_val:.4f} at step {best_step}")
    print(f"  Final val_loss: {final_val:.4f} at step {final_step}")

    # Overfitting detection
    if best_val > 0 and final_val > best_val * 1.2:
        pct = (final_val - best_val) / best_val * 100
        print(f"\n  ⚠️  OVERFITTING DETECTED: Final val_loss is {pct:.0f}% above best.")
    elif best_val == 0 and final_val > 0:
        print(f"\n  ⚠️  Best val_loss was 0.0; final val_loss is {final_val:.4f} — possible overfitting from a near-perfect early checkpoint.")
    elif final_train and final_val / final_train > 1.5:
        ratio = final_val / final_train
        print(f"\n  ⚠️  MODERATE OVERFITTING: val/train ratio = {ratio:.2f}")
    else:
        print(f"\n  ✅ Training looks healthy. No significant overfitting detected.")

    # List checkpoints and recommend best deployable one
    print(f"\n  Checkpoints:")
    available_checkpoints = []
    try:
        cps = client.fine_tuning.jobs.checkpoints.list(job_id)
        if cps.data:
            for cp in sorted(cps.data, key=lambda c: c.step_number):
                vl = cp.metrics.valid_loss if cp.metrics and cp.metrics.valid_loss is not None else None
                model_id = cp.fine_tuned_model_checkpoint or "N/A"
                vl_str = f"{vl:.4f}" if vl is not None else "N/A"
                available_checkpoints.append((cp.step_number, vl, model_id))
                print(f"    Step {cp.step_number}: val_loss={vl_str}, model={model_id}")
        else:
            print("    No checkpoints available.")
    except Exception as e:
        print(f"    Could not retrieve checkpoints: {e}")

    # Recommend the best deployable checkpoint
    if available_checkpoints and best_val > 0 and final_val > best_val * 1.2:
        # Find the checkpoint with the lowest val_loss, or nearest to best_step
        best_cp = None
        if any(vl is not None for _, vl, _ in available_checkpoints):
            # Use checkpoint with lowest val_loss
            scored_cps = [(s, vl, m) for s, vl, m in available_checkpoints if vl is not None]
            if scored_cps:
                best_cp = min(scored_cps, key=lambda x: x[1])
        else:
            # No val_loss on checkpoints — pick the one nearest to (but not exceeding) best_step
            earlier_cps = [(s, vl, m) for s, vl, m in available_checkpoints if s <= best_step]
            if earlier_cps:
                best_cp = max(earlier_cps, key=lambda x: x[0])
            elif available_checkpoints:
                best_cp = available_checkpoints[0]

        if best_cp:
            cp_step, cp_vl, cp_model = best_cp
            vl_info = f" (val_loss={cp_vl:.4f})" if cp_vl is not None else ""
            print(f"\n  🎯 Recommended checkpoint: step {cp_step}{vl_info}")
            print(f"     Model ID: {cp_model}")
            print(f"     (Best val_loss was at step {best_step}, nearest deployable checkpoint is step {cp_step})")
            print(f"     Alternatively, retrain with fewer epochs to avoid overfitting.")
        else:
            print(f"\n  Recommendation: Retrain with fewer epochs (best val_loss was at step {best_step}).")


def main():
    parser = HelpOnErrorParser(description="Analyze fine-tuning training curves")
    parser.add_argument("--base-url", default=os.environ.get("OPENAI_BASE_URL"),
                        help="Project /v1/ URL (preferred)")
    parser.add_argument("--endpoint", default=os.environ.get("AZURE_OPENAI_ENDPOINT"),
                        help="Azure OpenAI endpoint (fallback)")
    parser.add_argument("--project-endpoint", default=os.environ.get("AZURE_AI_PROJECT_ENDPOINT"),
                        help="Azure AI project endpoint (Foundry SDK)")
    parser.add_argument("--api-key", default=os.environ.get("AZURE_OPENAI_API_KEY"))
    parser.add_argument("--job-id", required=True, help="Fine-tuning job ID")
    parser.add_argument("--download-csv", help="Save results CSV to this path")
    args = parser.parse_args()

    client, method = get_clients(
        base_url=args.base_url, azure_endpoint=args.endpoint,
        project_endpoint=args.project_endpoint, api_key=args.api_key
    )
    analyze_job(client, args.job_id, args.download_csv)


if __name__ == "__main__":
    main()

# /// script
# dependencies = [
#   "openai>=1.0",
#   "azure-identity",
#   "azure-ai-projects",
# ]
# ///
"""
cleanup.py — Clean up fine-tuning resources to avoid quota exhaustion.

Lists and optionally deletes uploaded files and cancels pending jobs.
Useful after experimentation to reclaim quota (max 100 files per resource,
deployment slots are limited).

Usage:
  python cleanup.py --list                            # List all resources
  python cleanup.py --list --type files               # List only files
  python cleanup.py --delete-files --older-than 7     # Delete files older than 7 days
  python cleanup.py --delete-files --dry-run          # Preview what would be deleted
  python cleanup.py --cancel-pending                  # Cancel queued jobs
"""

import argparse
import os
import sys

try:
    sys.stdout.reconfigure(encoding="utf-8")
    sys.stderr.reconfigure(encoding="utf-8")
except (AttributeError, OSError):
    pass  # Stream not reconfigurable (older Python or non-tty); default encoding is fine
from datetime import datetime, timezone

sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from common import HelpOnErrorParser, get_clients


def list_deployments(client):
    """List fine-tuned model deployments. Returns deployment info from jobs."""
    deployments = []
    for job in _iter_all_jobs(client):
        if job.fine_tuned_model and job.status == "succeeded":
            deployments.append({
                "job_id": job.id,
                "model": job.fine_tuned_model,
                "base_model": job.model,
                "created": datetime.fromtimestamp(job.created_at, tz=timezone.utc),
                "tokens": job.trained_tokens,
            })
    return deployments


def _iter_all_jobs(client, page_size=100):
    """Yield every fine-tuning job, paginating through the API.

    The OpenAI/Azure SDK's `jobs.list(limit=N)` returns at most N jobs with no
    auto-paging. Users with >100 jobs would otherwise miss older jobs entirely.
    """
    after = None
    while True:
        kwargs = {"limit": page_size}
        if after:
            kwargs["after"] = after
        page = client.fine_tuning.jobs.list(**kwargs)
        items = list(page)
        if not items:
            break
        for job in items:
            yield job
        if len(items) < page_size:
            break
        # Cursor-based paging: use last job's id as `after`
        after = items[-1].id


def list_files(client):
    """List uploaded files."""
    files = client.files.list()
    result = []
    for f in files:
        result.append({
            "id": f.id,
            "filename": f.filename,
            "purpose": f.purpose,
            "bytes": f.bytes,
            "created": datetime.fromtimestamp(f.created_at, tz=timezone.utc),
            "status": f.status,
        })
    return result


def list_jobs(client):
    """List fine-tuning jobs."""
    result = []
    for job in _iter_all_jobs(client):
        result.append({
            "id": job.id,
            "status": job.status,
            "model": job.model,
            "fine_tuned_model": job.fine_tuned_model or "—",
            "created": datetime.fromtimestamp(job.created_at, tz=timezone.utc),
        })
    return result


def format_age(dt):
    """Format a datetime as a human-readable age."""
    delta = datetime.now(timezone.utc) - dt
    if delta.days > 0:
        return f"{delta.days}d ago"
    hours = delta.seconds // 3600
    return f"{hours}h ago"


def format_bytes(b):
    """Format bytes as human-readable size."""
    if not b:
        return "—"
    if b > 1_000_000:
        return f"{b/1_000_000:.1f} MB"
    if b > 1_000:
        return f"{b/1_000:.0f} KB"
    return f"{b} B"


def show_list(client, resource_type="all"):
    """Display current resources."""
    if resource_type in ("all", "jobs"):
        jobs = list_jobs(client)
        print(f"\n📋 Fine-tuning jobs ({len(jobs)}):")
        if jobs:
            print(f"  {'ID':<25} {'Status':<12} {'Model':<20} {'Age':<10}")
            print(f"  {'-'*25} {'-'*12} {'-'*20} {'-'*10}")
            for j in jobs:
                print(f"  {j['id'][:24]:<25} {j['status']:<12} {j['model']:<20} {format_age(j['created'])}")
        else:
            print("  (none)")

    if resource_type in ("all", "deployments"):
        deps = list_deployments(client)
        print(f"\n🚀 Fine-tuned models ({len(deps)}):")
        if deps:
            print(f"  {'Model':<60} {'Age':<10}")
            print(f"  {'-'*60} {'-'*10}")
            for d in deps:
                name = d['model'][:59]
                print(f"  {name:<60} {format_age(d['created'])}")
        else:
            print("  (none)")

    if resource_type in ("all", "files"):
        files = list_files(client)
        print(f"\n📁 Uploaded files ({len(files)}):")
        if files:
            print(f"  {'ID':<40} {'Size':>8} {'Purpose':<12} {'Age':<10} {'Status':<10}")
            print(f"  {'-'*40} {'-'*8} {'-'*12} {'-'*10} {'-'*10}")
            for f in files:
                print(f"  {f['id']:<40} {format_bytes(f['bytes']):>8} {f['purpose']:<12} {format_age(f['created']):<10} {f['status']}")
        else:
            print("  (none)")

        # Quota warning
        if len(files) >= 80:
            print(f"\n  ⚠️ {len(files)}/100 file slots used — approaching quota limit!")


def delete_files(client, older_than_days=None, dry_run=False):
    """Delete uploaded files, optionally filtering by age."""
    files = list_files(client)
    now = datetime.now(timezone.utc)

    to_delete = []
    for f in files:
        if older_than_days:
            age_days = (now - f["created"]).days
            if age_days < older_than_days:
                continue
        to_delete.append(f)

    if not to_delete:
        print("No files to delete.")
        return

    label = f"older than {older_than_days} days" if older_than_days else "all"
    print(f"\n{'[DRY RUN] ' if dry_run else ''}Deleting {len(to_delete)} files ({label}):")

    deleted = 0
    for f in to_delete:
        print(f"  {'Would delete' if dry_run else 'Deleting'}: {f['id']} ({f['filename']}, {format_age(f['created'])})")
        if not dry_run:
            try:
                client.files.delete(f["id"])
                deleted += 1
            except Exception as e:
                print(f"    ❌ Failed: {e}")

    if not dry_run:
        print(f"\n✅ Deleted {deleted}/{len(to_delete)} files")


def cancel_pending_jobs(client, dry_run=False):
    """Cancel any pending or queued jobs."""
    jobs = list_jobs(client)
    pending = [j for j in jobs if j["status"] in ("pending", "queued", "validating_files")]

    if not pending:
        print("No pending jobs to cancel.")
        return

    print(f"\n{'[DRY RUN] ' if dry_run else ''}Cancelling {len(pending)} pending jobs:")
    for j in pending:
        print(f"  {'Would cancel' if dry_run else 'Cancelling'}: {j['id']} ({j['status']})")
        if not dry_run:
            try:
                client.fine_tuning.jobs.cancel(j["id"])
            except Exception as e:
                print(f"    ❌ Failed: {e}")


def build_parser():
    parser = HelpOnErrorParser(
        description="Clean up fine-tuning resources (deployments, files, jobs)",
        epilog=(
            "Examples:\n"
            "  python cleanup.py --list                         # Show all resources\n"
            "  python cleanup.py --list --type files             # Show files only\n"
            "  python cleanup.py --delete-files --older-than 7   # Delete files older than 7 days\n"
            "  python cleanup.py --delete-files --dry-run        # Preview what would be deleted\n"
            "  python cleanup.py --cancel-pending                # Cancel queued jobs"
        ),
        formatter_class=argparse.RawTextHelpFormatter,
    )
    parser.add_argument("--base-url", default=os.environ.get("OPENAI_BASE_URL"), help="Project /v1/ endpoint URL")
    parser.add_argument("--endpoint", default=os.environ.get("AZURE_OPENAI_ENDPOINT"),
                        help="Azure OpenAI endpoint (fallback)")
    parser.add_argument("--api-key", default=os.environ.get("AZURE_OPENAI_API_KEY"), help="API key")
    parser.add_argument("--project-endpoint", default=os.environ.get("AZURE_AI_PROJECT_ENDPOINT"),
                        help="Azure AI project endpoint")

    parser.add_argument("--list", action="store_true", help="List resources")
    parser.add_argument("--type", choices=["all", "jobs", "deployments", "files"], default="all",
                        help="Resource type to list (default: all)")

    parser.add_argument("--delete-files", action="store_true", help="Delete uploaded files")
    parser.add_argument("--older-than", type=int, default=None,
                        help="Only delete files older than N days (use with --delete-files)")
    parser.add_argument("--cancel-pending", action="store_true", help="Cancel pending/queued jobs")
    parser.add_argument("--dry-run", action="store_true", help="Preview changes without executing")
    return parser


if __name__ == "__main__":
    parser = build_parser()
    if len(sys.argv) == 1:
        parser.print_help()
        sys.exit(0)

    args = parser.parse_args()
    client, method = get_clients(base_url=args.base_url, azure_endpoint=args.endpoint, project_endpoint=args.project_endpoint, api_key=args.api_key)

    if args.list:
        show_list(client, args.type)

    if args.delete_files:
        delete_files(client, older_than_days=args.older_than, dry_run=args.dry_run)

    if args.cancel_pending:
        cancel_pending_jobs(client, dry_run=args.dry_run)

"""
common.py — Shared Azure AI Foundry authentication and client setup.

Supports three connection methods in order of preference:
1. /v1/ project endpoint (simplest, preferred)
2. Foundry SDK with DefaultAzureCredential (no API key needed, cloud-native)
3. Azure OpenAI endpoint (classic)

AAD tokens are auto-refreshed via azure.identity for long-running scripts
(monitor_training.py, generate_distillation_data.py, etc.).

Usage:
    from common import get_clients, upload_file

    # Method 1: Project /v1/ endpoint (preferred)
    clients = get_clients(base_url="https://<resource>.services.ai.azure.com/api/projects/<project>/openai/v1/",
                          api_key="KEY")

    # Method 2: Foundry SDK (DefaultAzureCredential — no API key needed)
    clients = get_clients(project_endpoint="https://<resource>.services.ai.azure.com/api/projects/<project>")

    # Method 3: Azure OpenAI endpoint
    clients = get_clients(azure_endpoint="https://<resource>.openai.azure.com",
                          api_key="KEY")
"""
import argparse
import os
import sys



try:
    sys.stdout.reconfigure(encoding="utf-8")
    sys.stderr.reconfigure(encoding="utf-8")
except (AttributeError, OSError):
    pass  # Stream not reconfigurable (older Python or non-tty); default encoding is fine
_AZURE_COGSERVICES_SCOPE = "https://cognitiveservices.azure.com/.default"


def _clamp_score(v, default=0):
    """Clamp a judge score to [1, 10]. Returns `default` for missing/non-numeric values.

    LLM judges occasionally return out-of-range integers (e.g., 15) or non-numeric
    strings ("high"). Without clamping, these distort aggregate scores or crash
    `int()`. We use 0 as a sentinel for "missing/failed" so callers can filter via
    `score > 0`.
    """
    if v is None:
        return default
    try:
        return max(1, min(10, int(v)))
    except (ValueError, TypeError):
        return default


class HelpOnErrorParser(argparse.ArgumentParser):
    """ArgumentParser that prints full help when arguments are invalid.
    
    Standard ArgumentParser only prints a one-line usage summary on error,
    which isn't helpful for first-time users. This prints the full --help.
    """

    def error(self, message):
        self.print_help(sys.stderr)
        self.exit(2, f"\nerror: {message}\n")


def _make_token_provider():
    """Create an auto-refreshing AAD token provider for long-running scripts.
    
    Returns a callable that the OpenAI SDK calls before each request to get
    a fresh token. Tokens are cached and refreshed ~5 min before expiry.
    """
    from azure.identity import DefaultAzureCredential
    credential = DefaultAzureCredential()

    def get_token():
        try:
            token = credential.get_token(_AZURE_COGSERVICES_SCOPE)
            return token.token
        except Exception as e:
            raise RuntimeError(
                f"Azure AD authentication failed: {e}\n"
                "Ensure you're logged in (az login) or have valid "
                "AZURE_CLIENT_ID/AZURE_TENANT_ID/AZURE_CLIENT_SECRET set."
            ) from e

    return get_token


def get_clients(base_url=None, azure_endpoint=None, project_endpoint=None, api_key=None):
    """Initialize and return OpenAI-compatible client.

    Tries in order:
    1. Project /v1/ endpoint with openai.OpenAI() (simplest, preferred)
    2. Foundry SDK with AIProjectClient.get_openai_client() (no API key needed)
    3. Azure OpenAI endpoint with openai.AzureOpenAI() (classic)

    When using DefaultAzureCredential (no API key), tokens are auto-refreshed
    so long-running scripts won't fail with 401 after ~60 min.

    Returns: (openai_client, method_name)
    """
    # Method 1: /v1/ project endpoint
    base_url = base_url or os.environ.get("OPENAI_BASE_URL")
    api_key = api_key or os.environ.get("AZURE_OPENAI_API_KEY")

    if base_url:
        import openai
        if not api_key:
            try:
                token_provider = _make_token_provider()
                token_provider()  # verify it works
                # Use a custom httpx auth class that refreshes the token on each request
                import httpx

                class _AzureADAuth(httpx.Auth):
                    def __init__(self, provider):
                        self._provider = provider

                    def auth_flow(self, request):
                        request.headers["Authorization"] = f"Bearer {self._provider()}"
                        yield request

                client = openai.OpenAI(
                    base_url=base_url,
                    api_key="aad",  # required by SDK but overridden by auth
                    http_client=httpx.Client(auth=_AzureADAuth(token_provider)),
                )
                print(f"✅ Connected via /v1/ project endpoint (DefaultAzureCredential, auto-refresh)")
                return client, "project-v1-aad"
            except Exception as e:
                print(f"⚠️ No API key and DefaultAzureCredential failed: {e}")
        else:
            client = openai.OpenAI(base_url=base_url, api_key=api_key)
            print(f"✅ Connected via /v1/ project endpoint")
            return client, "project-v1"

    # Method 2: Foundry SDK
    project_endpoint = project_endpoint or os.environ.get("AZURE_AI_PROJECT_ENDPOINT")
    if project_endpoint:
        try:
            from azure.ai.projects import AIProjectClient
            from azure.identity import DefaultAzureCredential

            credential = DefaultAzureCredential()
            project_client = AIProjectClient(endpoint=project_endpoint, credential=credential)
            openai_client = project_client.get_openai_client()
            print(f"✅ Connected via Foundry SDK")
            return openai_client, "foundry-sdk"
        except Exception as e:
            print(f"⚠️ Foundry SDK failed: {e}")

    # Method 3: Azure OpenAI endpoint
    azure_endpoint = azure_endpoint or os.environ.get("AZURE_OPENAI_ENDPOINT")
    if azure_endpoint:
        import openai
        if api_key:
            client = openai.AzureOpenAI(
                azure_endpoint=azure_endpoint,
                api_key=api_key,
                api_version="2025-04-01-preview",
            )
            print(f"✅ Connected via Azure OpenAI endpoint")
            return client, "azure-openai"
        else:
            # No API key — use DefaultAzureCredential with auto-refresh
            try:
                token_provider = _make_token_provider()
                token_provider()  # verify it works
                client = openai.AzureOpenAI(
                    azure_endpoint=azure_endpoint,
                    azure_ad_token_provider=token_provider,
                    api_version="2025-04-01-preview",
                )
                print(f"✅ Connected via Azure OpenAI endpoint (DefaultAzureCredential, auto-refresh)")
                return client, "azure-openai-aad"
            except Exception as e:
                print(f"⚠️ DefaultAzureCredential failed for Azure endpoint: {e}")

    print("❌ No valid connection method. Set one of:")
    print("   OPENAI_BASE_URL (preferred)")
    print("   AZURE_AI_PROJECT_ENDPOINT (Foundry SDK)")
    print("   AZURE_OPENAI_ENDPOINT + AZURE_OPENAI_API_KEY")
    raise SystemExit(1)


def upload_file(openai_client, filepath: str, purpose: str = "fine-tune") -> str:
    """Upload a file to Azure AI Foundry and wait for processing."""
    print(f"📤 Uploading {filepath}...")
    with open(filepath, "rb") as f:
        file_obj = openai_client.files.create(file=f, purpose=purpose)
    print(f"   File ID: {file_obj.id}")
    print(f"   Waiting for processing...")
    openai_client.files.wait_for_processing(file_obj.id)
    print(f"   ✅ File ready")
    return file_obj.id

# /// script
# dependencies = [
#   "openai>=1.0",
# ]
# ///
"""
convert_dataset.py — Convert between SFT, DPO, and RFT dataset formats.

Usage:
  # Parquet/CSV to SFT JSONL
  python convert_dataset.py --input data.parquet --output train.jsonl --format sft \
      --user-column prompt --assistant-column response --system-prompt "You are helpful."

  # SFT JSONL to DPO (generates rejected via base model)
  python convert_dataset.py --input train.jsonl --output dpo.jsonl --format dpo \
      --base-model gpt-4.1-mini --endpoint $ENDPOINT --api-key $KEY

  # SFT JSONL to RFT JSONL (passthrough — same format, different intent)
  python convert_dataset.py --input train.jsonl --output rft.jsonl --format rft

  # DPO JSONL to SFT (extract chosen responses)
  python convert_dataset.py --input dpo.jsonl --output sft.jsonl --format sft-from-dpo
"""

import json
import os
import sys

try:
    sys.stdout.reconfigure(encoding="utf-8")
    sys.stderr.reconfigure(encoding="utf-8")
except (AttributeError, OSError):
    pass  # Stream not reconfigurable (older Python or non-tty); default encoding is fine
import time
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from common import HelpOnErrorParser, get_clients


def parquet_to_sft(input_path, output_path, user_col, assistant_col, system_prompt=None):
    """Convert a parquet or CSV file to SFT JSONL."""
    try:
        import pandas as pd
    except ImportError:
        print("Error: pandas required. Install with: pip install pandas pyarrow")
        sys.exit(1)

    if input_path.endswith(".parquet"):
        df = pd.read_parquet(input_path)
    elif input_path.endswith(".csv"):
        df = pd.read_csv(input_path)
    elif input_path.endswith(".json"):
        df = pd.read_json(input_path)
    else:
        print(f"Unsupported format: {input_path}. Use .parquet, .csv, or .json")
        sys.exit(1)

    if user_col not in df.columns or assistant_col not in df.columns:
        print(f"Error: Columns '{user_col}' and/or '{assistant_col}' not found.")
        print(f"Available columns: {list(df.columns)}")
        sys.exit(1)

    count = 0
    with open(output_path, "w", encoding="utf-8") as f:
        for _, row in df.iterrows():
            user_content = str(row[user_col]).strip()
            asst_content = str(row[assistant_col]).strip()
            if not user_content or not asst_content:
                continue

            messages = []
            if system_prompt:
                messages.append({"role": "system", "content": system_prompt})
            messages.append({"role": "user", "content": user_content})
            messages.append({"role": "assistant", "content": asst_content})

            f.write(json.dumps({"messages": messages}, ensure_ascii=False) + "\n")
            count += 1

    print(f"Converted {count} examples to SFT JSONL → {output_path}")


def sft_to_dpo(input_path, output_path, client, base_model):
    """Convert SFT to DPO by generating non-preferred responses from a base model.

    DPO format uses: input (system+user messages), preferred_output, non_preferred_output.
    """
    with open(input_path, encoding="utf-8") as inf:
        examples = []
        for ln, raw in enumerate(inf, 1):
            if not raw.strip():
                continue
            try:
                examples.append(json.loads(raw))
            except json.JSONDecodeError as e:
                print(f"  ⚠️ Skipping malformed JSON on line {ln}: {e}")
    count = 0

    with open(output_path, "w", encoding="utf-8") as f:
        for i, ex in enumerate(examples):
            msgs = ex["messages"]
            system_msgs = [m for m in msgs if m["role"] == "system"]
            user_msg = next((m for m in msgs if m["role"] == "user"), None)
            asst_msg = next((m for m in msgs if m["role"] == "assistant"), None)
            if not user_msg or not asst_msg:
                continue

            # Generate a non-preferred response from the base model
            try:
                gen_msgs = system_msgs + [user_msg]
                resp = client.chat.completions.create(
                    model=base_model,
                    messages=gen_msgs,
                    temperature=1.0,  # High temp for diversity
                    max_completion_tokens=2048,
                )
                rejected_content = resp.choices[0].message.content
            except Exception as e:
                print(f"  Skipping example {i}: {e}")
                continue

            if not rejected_content:
                # None or empty — content filter, finish=length with no text, etc.
                # Skip rather than emit a DPO entry with null content (trainer rejects).
                print(f"  Skipping example {i}: base model returned no content")
                continue

            # Build DPO entry with correct format
            input_messages = system_msgs + [user_msg]
            dpo_entry = {
                "input": {"messages": input_messages},
                "preferred_output": [asst_msg],
                "non_preferred_output": [{"role": "assistant", "content": rejected_content}],
            }
            f.write(json.dumps(dpo_entry, ensure_ascii=False) + "\n")
            count += 1

            if (i + 1) % 50 == 0:
                print(f"  Processed {i+1}/{len(examples)}")
                time.sleep(1)

    print(f"Converted {count} examples to DPO JSONL → {output_path}")


def sft_to_rft(input_path, output_path):
    """Convert SFT to RFT format.

    Strips assistant messages (RFT last message must be user) and adds a
    placeholder grader field. The user must populate grader reference fields
    (e.g., expected_answer) before training.
    """
    count = 0
    skipped = 0
    with open(output_path, "w", encoding="utf-8") as out:
        with open(input_path, encoding="utf-8") as inf:
            for ln, line in enumerate(inf, 1):
                if not line.strip():
                    continue
                try:
                    ex = json.loads(line)
                except json.JSONDecodeError as e:
                    print(f"  ⚠️ Skipping malformed JSON on line {ln}: {e}")
                    skipped += 1
                    continue
                msgs = ex.get("messages", [])
                # Keep only system + user messages; RFT last message must be user
                rft_msgs = [m for m in msgs if m["role"] in ("system", "user")]
                if not rft_msgs or rft_msgs[-1]["role"] != "user":
                    skipped += 1
                    continue
                # Extract assistant content as a reference answer placeholder
                asst_msgs = [m for m in msgs if m["role"] == "assistant"]
                expected = asst_msgs[-1]["content"] if asst_msgs else ""
                rft_entry = {"messages": rft_msgs, "expected_answer": expected}
                out.write(json.dumps(rft_entry, ensure_ascii=False) + "\n")
                count += 1
    print(f"Converted {count} examples to RFT JSONL → {output_path}")
    if skipped:
        print(f"  Skipped {skipped} examples (no user message)")
    print("Note: Review 'expected_answer' fields and update your grader to use item.expected_answer.")


def dpo_to_sft(input_path, output_path, system_prompt=None):
    """Extract chosen responses from DPO format to SFT format."""
    count = 0
    with open(output_path, "w", encoding="utf-8") as f:
        with open(input_path, encoding="utf-8") as inf:
            for ln, line in enumerate(inf, 1):
                if not line.strip():
                    continue
                try:
                    ex = json.loads(line)
                except json.JSONDecodeError as e:
                    print(f"  ⚠️ Skipping malformed JSON on line {ln}: {e}")
                    continue
                input_messages = ex["input"]["messages"]
                chosen_messages = ex["preferred_output"]

                messages = []
                if system_prompt:
                    messages.append({"role": "system", "content": system_prompt})
                    messages.extend(m for m in input_messages if m["role"] != "system")
                else:
                    messages.extend(input_messages)
                messages.extend(chosen_messages)
                f.write(json.dumps({"messages": messages}, ensure_ascii=False) + "\n")
                count += 1
    print(f"Extracted {count} chosen examples to SFT JSONL → {output_path}")


def main():
    parser = HelpOnErrorParser(description="Convert between fine-tuning dataset formats")
    parser.add_argument("--input", required=True, help="Input file path")
    parser.add_argument("--output", required=True, help="Output file path")
    parser.add_argument("--format", required=True,
                        choices=["sft", "dpo", "rft", "sft-from-dpo"],
                        help="Target format")

    # SFT from raw data
    parser.add_argument("--user-column", default="prompt", help="Column name for user input")
    parser.add_argument("--assistant-column", default="response", help="Column name for assistant output")
    parser.add_argument("--system-prompt", default=None, help="System prompt to prepend")

    # DPO generation (needs API connection)
    parser.add_argument("--base-url", default=os.environ.get("OPENAI_BASE_URL"),
                        help="Project /v1/ URL (preferred)")
    parser.add_argument("--endpoint", default=os.environ.get("AZURE_OPENAI_ENDPOINT"),
                        help="Azure OpenAI endpoint (fallback)")
    parser.add_argument("--project-endpoint", default=os.environ.get("AZURE_AI_PROJECT_ENDPOINT"),
                        help="Azure AI project endpoint (Foundry SDK)")
    parser.add_argument("--api-key", default=os.environ.get("AZURE_OPENAI_API_KEY"))
    parser.add_argument("--base-model", default="gpt-4.1-mini", help="Base model for generating rejections")

    args = parser.parse_args()

    if args.format == "sft":
        if args.input.endswith(".jsonl"):
            print("Input is already JSONL — assuming SFT format. Nothing to convert.")
            if args.input != args.output:
                import shutil
                shutil.copy2(args.input, args.output)
        else:
            parquet_to_sft(args.input, args.output, args.user_column,
                           args.assistant_column, args.system_prompt)

    elif args.format == "dpo":
        client, method = get_clients(
            base_url=args.base_url, azure_endpoint=args.endpoint,
            project_endpoint=args.project_endpoint, api_key=args.api_key
        )
        sft_to_dpo(args.input, args.output, client, args.base_model)

    elif args.format == "rft":
        sft_to_rft(args.input, args.output)

    elif args.format == "sft-from-dpo":
        dpo_to_sft(args.input, args.output, args.system_prompt)


if __name__ == "__main__":
    main()

# /// script
# dependencies = [
#   "openai>=1.0",
#   "requests",
#   "azure-identity",
# ]
# ///
"""
deploy_model.py — Deploy fine-tuned models on Azure AI Foundry via ARM REST API.

Supports all model families with correct format/SKU mapping.

Usage:
  python deploy_model.py --model-id "ft:gpt-4.1-mini-2025-04-14:..." --name "my-ft-eval" --capacity 100
  python deploy_model.py --model-id "ft:gpt-oss-20b:..." --name "oss-eval" --format Microsoft --sku GlobalStandard
  python deploy_model.py --delete --name "my-ft-eval"
  python deploy_model.py --list
"""

import os
import subprocess
import sys

try:
    sys.stdout.reconfigure(encoding="utf-8")
    sys.stderr.reconfigure(encoding="utf-8")
except (AttributeError, OSError):
    pass  # Stream not reconfigurable (older Python or non-tty); default encoding is fine
import time
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from common import HelpOnErrorParser

import requests


def _safe_error_msg(resp):
    """Extract error message from response, handling non-JSON bodies (HTML 502/503)."""
    try:
        return resp.json().get("error", {}).get("message", resp.text[:200])
    except (ValueError, KeyError):
        return resp.text[:200] if resp.text else "Unknown error"

# Default Azure resource coordinates — override with env vars or args
DEFAULT_SUB = os.environ.get("AZURE_SUBSCRIPTION_ID", "")
DEFAULT_RG = os.environ.get("AZURE_RESOURCE_GROUP", "")
DEFAULT_ACCOUNT = os.environ.get("AZURE_COGSERVICES_ACCOUNT", "")
AZ_CLI = os.environ.get("AZ_CLI_PATH")
if not AZ_CLI:
    import shutil
    AZ_CLI = shutil.which("az")
    if not AZ_CLI:
        # Common Windows paths
        for candidate in [
            r"C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\wbin\az.cmd",
            r"C:\Program Files\Microsoft SDKs\Azure\CLI2\wbin\az.cmd",
        ]:
            if os.path.exists(candidate):
                AZ_CLI = candidate
                break
    if not AZ_CLI:
        AZ_CLI = "az"  # last resort, hope it's on PATH

# Model format auto-detection rules
FORMAT_RULES = [
    (lambda m: "oss-20b" in m.lower() or "oss20b" in m.lower(), "Microsoft", "GlobalStandard"),
    (lambda m: "ministral" in m.lower() or "mistral" in m.lower(), "Mistral AI", "GlobalStandard"),
    (lambda m: "llama" in m.lower() or "meta" in m.lower(), "Meta", "GlobalStandard"),
    (lambda m: "qwen" in m.lower() or "alibaba" in m.lower(), "Alibaba", "GlobalStandard"),
    (lambda m: True, "OpenAI", "Standard"),  # Default fallback
]


def get_arm_token():
    """Get a fresh ARM token from Azure CLI."""
    result = subprocess.run(
        [AZ_CLI, "account", "get-access-token", "--query", "accessToken", "-o", "tsv"],
        capture_output=True, text=True,
    )
    token = result.stdout.strip()
    if not token:
        raise RuntimeError(f"Failed to get ARM token: {result.stderr}")
    return token


def arm_url(sub, rg, account, deploy_name=None):
    """Build the ARM REST API URL."""
    base = (f"https://management.azure.com/subscriptions/{sub}"
            f"/resourceGroups/{rg}"
            f"/providers/Microsoft.CognitiveServices/accounts/{account}"
            f"/deployments")
    if deploy_name:
        base += f"/{deploy_name}"
    return base + "?api-version=2024-10-01"


def detect_format(model_id):
    """Auto-detect model format and SKU from model ID."""
    for check, fmt, sku in FORMAT_RULES:
        if check(model_id):
            return fmt, sku
    return "OpenAI", "Standard"


def create_deployment(sub, rg, account, name, model_id, model_format, sku, capacity):
    """Create a deployment via ARM REST API."""
    token = get_arm_token()
    url = arm_url(sub, rg, account, name)

    body = {
        "sku": {"name": sku, "capacity": capacity},
        "properties": {
            "model": {
                "format": model_format,
                "name": model_id,
                "version": "1",
            }
        },
    }

    resp = requests.put(url, headers={
        "Authorization": f"Bearer {token}",
        "Content-Type": "application/json",
    }, json=body, timeout=(10, 120))

    if resp.status_code in (200, 201):
        print(f"✅ Deployment '{name}' created (format={model_format}, sku={sku}, capacity={capacity})")
        return True
    else:
        print(f"❌ Deployment failed ({resp.status_code}): {_safe_error_msg(resp)}")
        return False


def wait_for_deployment(sub, rg, account, name, timeout=600, poll_interval=15):
    """Wait for deployment to reach 'Succeeded' state."""
    url = arm_url(sub, rg, account, name)
    start = time.time()

    while time.time() - start < timeout:
        token = get_arm_token()
        try:
            resp = requests.get(url, headers={"Authorization": f"Bearer {token}"}, timeout=(10, 60))
        except requests.exceptions.RequestException as e:
            print(f"  ⚠️ Polling error: {e} — retrying in {poll_interval}s")
            time.sleep(poll_interval)
            continue
        if resp.status_code == 200:
            try:
                state = resp.json().get("properties", {}).get("provisioningState", "Unknown")
            except (ValueError, KeyError):
                state = "Unknown"
            print(f"  Status: {state}")
            if state == "Succeeded":
                return True
            if state in ("Failed", "Canceled"):
                print(f"  Deployment {state}.")
                return False
        time.sleep(poll_interval)

    print(f"  Timed out after {timeout}s")
    return False


def delete_deployment(sub, rg, account, name):
    """Delete a deployment."""
    token = get_arm_token()
    url = arm_url(sub, rg, account, name)
    resp = requests.delete(url, headers={"Authorization": f"Bearer {token}"}, timeout=(10, 60))
    if resp.status_code in (200, 202, 204):
        print(f"✅ Deployment '{name}' deleted.")
    else:
        print(f"❌ Delete failed ({resp.status_code}): {_safe_error_msg(resp)}")


def list_deployments(sub, rg, account):
    """List all deployments."""
    token = get_arm_token()
    url = arm_url(sub, rg, account)
    resp = requests.get(url, headers={"Authorization": f"Bearer {token}"}, timeout=(10, 60))
    if resp.status_code != 200:
        print(f"❌ Failed to list deployments ({resp.status_code}): {_safe_error_msg(resp)}")
        return

    try:
        deployments = resp.json().get("value", [])
    except (ValueError, KeyError):
        print(f"❌ Failed to parse deployment list: {resp.text[:200]}")
        return
    if not deployments:
        print("No deployments found.")
        return

    print(f"{'Name':<40} {'Model':<40} {'SKU':<15} {'State':<15}")
    print("─" * 110)
    for d in deployments:
        name = d.get("name", "?")
        model = d.get("properties", {}).get("model", {}).get("name", "?")
        sku = d.get("sku", {}).get("name", "?")
        state = d.get("properties", {}).get("provisioningState", "?")
        print(f"{name:<40} {model:<40} {sku:<15} {state:<15}")


def main():
    parser = HelpOnErrorParser(description="Deploy fine-tuned models on Azure AI Foundry")
    parser.add_argument("--sub", default=DEFAULT_SUB, help="Azure subscription ID")
    parser.add_argument("--rg", default=DEFAULT_RG, help="Resource group")
    parser.add_argument("--account", default=DEFAULT_ACCOUNT, help="Cognitive Services account")

    # Actions
    parser.add_argument("--list", action="store_true", help="List all deployments")
    parser.add_argument("--delete", action="store_true", help="Delete a deployment")
    parser.add_argument("--wait", action="store_true", help="Wait for deployment to succeed")

    # Deployment config
    parser.add_argument("--name", help="Deployment name (max 64 chars, alphanumeric + hyphens)")
    parser.add_argument("--model-id", help="Fine-tuned model ID (e.g., ft:gpt-4.1-mini:...)")
    parser.add_argument("--format", help="Model format (auto-detected if not specified)")
    parser.add_argument("--sku", help="SKU name (auto-detected if not specified)")
    parser.add_argument("--capacity", type=int, default=100, help="TPM capacity in thousands")

    args = parser.parse_args()

    if not all([args.sub, args.rg, args.account]):
        print("Error: Set --sub/--rg/--account or AZURE_SUBSCRIPTION_ID/AZURE_RESOURCE_GROUP/AZURE_COGSERVICES_ACCOUNT")
        sys.exit(1)

    if args.list:
        list_deployments(args.sub, args.rg, args.account)
        return

    if not args.name:
        print("Error: --name required for create/delete/wait")
        sys.exit(1)

    if args.delete:
        delete_deployment(args.sub, args.rg, args.account, args.name)
        return

    if args.wait and not args.model_id:
        # Wait-only mode: poll an existing deployment
        success = wait_for_deployment(args.sub, args.rg, args.account, args.name)
        sys.exit(0 if success else 1)

    if not args.model_id:
        print("Error: --model-id required for create")
        sys.exit(1)

    # Auto-detect format/SKU if not specified
    model_format = args.format
    sku = args.sku
    if not model_format or not sku:
        auto_fmt, auto_sku = detect_format(args.model_id)
        model_format = model_format or auto_fmt
        sku = sku or auto_sku
        print(f"Auto-detected: format={model_format}, sku={sku}")

    created = create_deployment(args.sub, args.rg, args.account, args.name,
                      args.model_id, model_format, sku, args.capacity)

    if args.wait and created:
        wait_for_deployment(args.sub, args.rg, args.account, args.name)


if __name__ == "__main__":
    main()

# /// script
# dependencies = [
#   "openai>=1.0",
#   "azure-identity",
# ]
# ///
"""
evaluate_model.py — Custom 2-dimension LLM judge evaluator for fine-tuned models.

This is a lightweight evaluation script using the OpenAI API directly.
For production evaluation, prefer the Azure AI Evaluation SDK which provides
built-in graders, batch evaluation, and guardrail metrics. See
references/evaluation.md for SDK patterns.

Uses the OpenAI API directly to:
1. Generate responses from a deployed fine-tuned model
2. Grade each response on correctness and conciseness using an LLM judge
3. Produce aggregate quality scores (weighted 70% correctness, 30% conciseness)

By default, system prompts from each test example's messages array are used
during generation. The --system-prompt flag overrides this for all examples.

Usage:
  python evaluate_model.py \
      --deployment-name my-ft-eval \
      --test-file test.jsonl \
      --judge-model gpt-4o \
      --output results.json

  python evaluate_model.py \
      --base-url "$BASE_URL" --api-key "$API_KEY" \
      --deployment-name my-ft-eval \
      --test-file test.jsonl \
      --concurrency 4
"""

import json
import os
import re
import sys

try:
    sys.stdout.reconfigure(encoding="utf-8")
    sys.stderr.reconfigure(encoding="utf-8")
except (AttributeError, OSError):
    pass  # Stream not reconfigurable (older Python or non-tty); default encoding is fine
import time
from concurrent.futures import ThreadPoolExecutor, as_completed
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from common import HelpOnErrorParser, get_clients, _clamp_score


JUDGE_PROMPT = """You are evaluating the quality of a model's output for a given task.

## Task prompt
{prompt}

## Reference answer
{reference}

## Model output
{output}

## Scoring

Rate the output on two dimensions, each on a scale of 1-10:

**Correctness** (1-10): Does the output correctly accomplish the task?
- 1-3: Fundamentally wrong or broken
- 4-6: Partially correct with significant issues
- 7-8: Mostly correct with minor issues
- 9-10: Fully correct

**Conciseness** (1-10): Is the output appropriately concise?
- 1-3: Extremely verbose or padded
- 4-6: Contains unnecessary content
- 7-8: Mostly concise with minor excess
- 9-10: Clean and focused

Return ONLY a JSON object: {{"correctness": <int>, "conciseness": <int>}}"""


def load_test_data(filepath):
    """Load held-out test set. Expects JSONL with 'messages' array.

    Extracts the system prompt (if present), user prompt, and assistant
    reference from each example so per-example system prompts are preserved.
    """
    data = []
    with open(filepath, encoding="utf-8") as f:
        for i, line in enumerate(f):
            if not line.strip():
                continue
            try:
                ex = json.loads(line)
            except json.JSONDecodeError as e:
                print(f"⚠️ Skipping malformed JSON on line {i+1}: {e}")
                continue
            msgs = ex.get("messages")
            if not isinstance(msgs, list):
                print(f"⚠️ Skipping example {i}: missing or invalid 'messages' list")
                continue
            prompt = next((m["content"] for m in msgs if m["role"] == "user"), None)
            reference = next((m["content"] for m in msgs if m["role"] == "assistant"), None)
            if not prompt:
                print(f"⚠️ Skipping example {i}: missing 'user' message")
                continue
            if not reference:
                print(f"⚠️ Skipping example {i}: missing 'assistant' message")
                continue
            system_msgs = [m["content"] for m in msgs if m["role"] == "system"]
            system_prompt = system_msgs[0] if system_msgs else None
            data.append({"prompt": prompt, "reference": reference, "system_prompt": system_prompt})
    return data


def generate_response(client, deployment, prompt, system_prompt=None, max_retries=3):
    """Generate a single response from the deployed model."""
    messages = []
    if system_prompt:
        messages.append({"role": "system", "content": system_prompt})
    messages.append({"role": "user", "content": prompt})

    for attempt in range(max_retries):
        try:
            resp = client.chat.completions.create(
                model=deployment,
                messages=messages,
                temperature=0.0,
                max_completion_tokens=2048,
            )
            content = resp.choices[0].message.content
            if content is None:
                # Content filter or empty completion — surface as an error sentinel
                # so the aggregate filter at line ~`.startswith("ERROR:")` skips it.
                finish = getattr(resp.choices[0], "finish_reason", "unknown")
                return f"ERROR: empty content (finish_reason={finish})"
            return content
        except Exception as e:
            if attempt >= max_retries - 1:
                return f"ERROR: {e}"
            time.sleep(3 * (attempt + 1))
    return "ERROR: max retries exceeded"


def grade_response(judge_client, judge_model, prompt, reference, output, max_retries=3):
    """Grade a response using the LLM judge."""
    judge_input = JUDGE_PROMPT.format(prompt=prompt, reference=reference, output=output)

    for attempt in range(max_retries):
        try:
            resp = judge_client.chat.completions.create(
                model=judge_model,
                messages=[{"role": "user", "content": judge_input}],
                temperature=0.0,
                max_completion_tokens=200,
            )
            text = (resp.choices[0].message.content or "").strip()
            # Extract JSON from response
            match = re.search(r'\{[^}]+\}', text)
            if match:
                scores = json.loads(match.group())
                return {
                    "correctness": _clamp_score(scores.get("correctness")),
                    "conciseness": _clamp_score(scores.get("conciseness")),
                }
        except Exception as e:
            if attempt < max_retries - 1:
                time.sleep(2)
            else:
                return {"correctness": 0, "conciseness": 0, "error": str(e)}

    return {"correctness": 0, "conciseness": 0, "error": "All retries failed"}


def main():
    parser = HelpOnErrorParser(description="Evaluate a fine-tuned model with LLM judge")
    parser.add_argument("--base-url", default=os.environ.get("OPENAI_BASE_URL"),
                        help="Project /v1/ URL (preferred)")
    parser.add_argument("--endpoint", default=os.environ.get("AZURE_OPENAI_ENDPOINT"),
                        help="Azure OpenAI endpoint (fallback)")
    parser.add_argument("--project-endpoint", default=os.environ.get("AZURE_AI_PROJECT_ENDPOINT"),
                        help="Azure AI project endpoint (Foundry SDK)")
    parser.add_argument("--api-key", default=os.environ.get("AZURE_OPENAI_API_KEY"))
    parser.add_argument("--deployment-name", required=True, help="Deployed model name")
    parser.add_argument("--test-file", required=True, help="Held-out test set (JSONL)")
    parser.add_argument("--system-prompt", default=None,
                        help="Override system prompt for all examples (default: use per-example system prompt from test data)")

    # Judge config
    parser.add_argument("--judge-model", default="gpt-4o", help="Model for LLM judge")
    parser.add_argument("--judge-endpoint", help="Endpoint for judge (default: same as model)")
    parser.add_argument("--judge-api-key", help="API key for judge (default: same as model)")

    # Output
    parser.add_argument("--output", default="eval_results.json", help="Output file")
    parser.add_argument("--concurrency", type=int, default=1,
                        help="Parallel grading workers (generation is always sequential)")

    args = parser.parse_args()

    # Set up model client via shared auth (supports /v1/, Foundry SDK, AzureOpenAI)
    model_client, method = get_clients(
        base_url=args.base_url, azure_endpoint=args.endpoint,
        project_endpoint=args.project_endpoint, api_key=args.api_key
    )

    # Set up judge client (defaults to same connection as model)
    judge_key = args.judge_api_key or args.api_key
    if args.judge_endpoint:
        judge_client, _ = get_clients(azure_endpoint=args.judge_endpoint, api_key=judge_key)
    elif args.judge_api_key:
        # Different API key but same endpoint — create a new client with the judge key
        judge_client, _ = get_clients(
            base_url=args.base_url, azure_endpoint=args.endpoint,
            project_endpoint=args.project_endpoint, api_key=judge_key
        )
    else:
        judge_client = model_client

    # Load data
    test_data = load_test_data(args.test_file)
    print(f"Loaded {len(test_data)} test examples from {args.test_file}")

    # Phase 1: Generate responses (sequential to avoid rate limits)
    print(f"\nGenerating responses from {args.deployment_name}...")
    for i, ex in enumerate(test_data):
        # Use CLI override if provided, otherwise use per-example system prompt
        effective_system_prompt = args.system_prompt if args.system_prompt is not None else ex.get("system_prompt")
        ex["output"] = generate_response(
            model_client, args.deployment_name, ex["prompt"], effective_system_prompt
        )
        if (i + 1) % 10 == 0:
            print(f"  Generated {i+1}/{len(test_data)}")

    errors = sum(1 for ex in test_data if ex["output"].startswith("ERROR:"))
    print(f"  Done. {errors} errors out of {len(test_data)}.")

    # Phase 2: Grade responses (parallel)
    print(f"\nGrading with {args.judge_model} (concurrency={args.concurrency})...")

    def grade_one(ex):
        return grade_response(judge_client, args.judge_model,
                              ex["prompt"], ex["reference"], ex["output"])

    with ThreadPoolExecutor(max_workers=args.concurrency) as pool:
        futures = {pool.submit(grade_one, ex): i for i, ex in enumerate(test_data)}
        for future in as_completed(futures):
            idx = futures[future]
            test_data[idx]["scores"] = future.result()

    # Aggregate
    valid_scores = [ex["scores"] for ex in test_data
                    if ex["scores"]["correctness"] > 0]
    if not valid_scores:
        print("No valid scores — all grading failed.")
        sys.exit(1)

    avg_corr = sum(s["correctness"] for s in valid_scores) / len(valid_scores)
    avg_conc = sum(s["conciseness"] for s in valid_scores) / len(valid_scores)
    combined = 0.7 * avg_corr + 0.3 * avg_conc

    print(f"\n{'='*50}")
    print(f"Results for {args.deployment_name}")
    print(f"  Correctness:  {avg_corr:.2f}")
    print(f"  Conciseness:  {avg_conc:.2f}")
    print(f"  Combined:     {combined:.2f}")
    print(f"  (N={len(valid_scores)} scored, {len(test_data)-len(valid_scores)} failed)")
    print(f"{'='*50}")

    # Save
    results = {
        "deployment": args.deployment_name,
        "judge_model": args.judge_model,
        "n_examples": len(test_data),
        "n_scored": len(valid_scores),
        "correctness": round(avg_corr, 2),
        "conciseness": round(avg_conc, 2),
        "combined": round(combined, 2),
        "details": [
            {
                "prompt": ex["prompt"][:200],
                "scores": ex.get("scores", {}),
            }
            for ex in test_data
        ],
    }

    with open(args.output, "w", encoding="utf-8") as f:
        json.dump(results, f, indent=2)
    print(f"\nDetailed results saved to {args.output}")


if __name__ == "__main__":
    main()

# /// script
# dependencies = [
#   "openai>=1.0",
#   "azure-identity",
# ]
# ///
"""
generate_distillation_data.py — Generate training data from a teacher model for distillation.

Creates a synthetic SFT dataset by:
1. Generating diverse prompts from combinatorial axes (topics × formats × contexts)
2. Having the teacher model produce responses
3. Quality-grading each response with an LLM judge
4. Filtering low-quality examples
5. Splitting into train/val/test JSONL files

Usage:
  python generate_distillation_data.py \
      --teacher gpt-4.1-mini \
      --system-prompt "You are a formal business writer." \
      --topics "earnings,risk,compliance" \
      --num-prompts 300 \
      --min-score 7.0 \
      --output-dir ./my_dataset

  # Or with a prompts file (one prompt per line):
  python generate_distillation_data.py \
      --teacher gpt-4.1-mini \
      --prompts-file my_prompts.txt \
      --output-dir ./my_dataset
"""

import json
import os
import random
import re
import sys

try:
    sys.stdout.reconfigure(encoding="utf-8")
    sys.stderr.reconfigure(encoding="utf-8")
except (AttributeError, OSError):
    pass  # Stream not reconfigurable (older Python or non-tty); default encoding is fine
import time
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from common import HelpOnErrorParser, get_clients, _clamp_score

import openai


def verify_deployment(client, model):
    """Verify a model deployment exists by sending a trivial request."""
    try:
        client.chat.completions.create(
            model=model,
            messages=[{"role": "user", "content": "Hi"}],
            max_completion_tokens=1,
        )
        return True
    except openai.NotFoundError:
        return False
    except Exception:
        return True  # other errors (rate limit, etc.) mean the deployment exists


def generate_combinatorial_prompts(topics, formats, contexts, n):
    """Generate diverse prompts from combinatorial axes."""
    prompts = []
    for _ in range(n):
        t = random.choice(topics)
        f = random.choice(formats)
        c = random.choice(contexts)
        prompts.append(f"Context: {c}\n\nWrite {f} about: {t}.")
    return prompts


def teacher_generate(client, model, system_prompt, prompt, retries=3):
    """Generate a single response from the teacher."""
    for attempt in range(retries):
        try:
            resp = client.chat.completions.create(
                model=model,
                messages=[
                    {"role": "system", "content": system_prompt},
                    {"role": "user", "content": prompt},
                ],
                temperature=0.7,
                max_completion_tokens=1024,
            )
            return resp.choices[0].message.content
        except Exception as e:
            if attempt >= retries - 1:
                print(f"  Failed after {retries} attempts: {e}")
                return None
            time.sleep(2 * (attempt + 1))
    return None


QUALITY_PROMPT = """Rate this AI-generated text on quality dimensions (1-10 each).

## Text to evaluate
{output}

## Dimensions
**Accuracy** (1-10): Is the content factually sound and coherent?
**Quality** (1-10): Is it well-written, clear, and professional?
**Task-fit** (1-10): Does it match the requested format and purpose?

Return ONLY JSON: {{"accuracy": <int>, "quality": <int>, "task_fit": <int>}}"""


def grade_output(client, judge_model, output, retries=3):
    for attempt in range(retries):
        try:
            resp = client.chat.completions.create(
                model=judge_model,
                messages=[{"role": "user", "content": QUALITY_PROMPT.format(output=output)}],
                temperature=0.0,
                max_completion_tokens=100,
            )
            text = (resp.choices[0].message.content or "").strip()
            match = re.search(r'\{[^}]+\}', text)
            if match:
                scores = json.loads(match.group())
                return {k: _clamp_score(v) for k, v in scores.items()}
        except Exception:
            if attempt < retries - 1:
                time.sleep(2)
    return None


def main():
    parser = HelpOnErrorParser(description="Generate distillation training data from a teacher model")
    parser.add_argument("--base-url", default=os.environ.get("OPENAI_BASE_URL"),
                        help="Project /v1/ URL (preferred)")
    parser.add_argument("--endpoint", default=os.environ.get("AZURE_OPENAI_ENDPOINT"),
                        help="Azure OpenAI endpoint (fallback)")
    parser.add_argument("--project-endpoint", default=os.environ.get("AZURE_AI_PROJECT_ENDPOINT"),
                        help="Azure AI project endpoint (Foundry SDK)")
    parser.add_argument("--api-key", default=os.environ.get("AZURE_OPENAI_API_KEY"))
    parser.add_argument("--teacher", required=True, help="Teacher model deployment name")
    parser.add_argument("--judge", default=None, help="Judge model (default: same as teacher)")
    parser.add_argument("--system-prompt", default="You are a helpful assistant.", help="System prompt for teacher")

    # Prompt generation (either combinatorial or from file)
    parser.add_argument("--prompts-file", help="File with one prompt per line (skips combinatorial generation)")
    parser.add_argument("--topics", help="Comma-separated topics for combinatorial prompts")
    parser.add_argument("--formats", default="a concise response,a brief summary,a detailed explanation",
                        help="Comma-separated output formats")
    parser.add_argument("--contexts", default="", help="Comma-separated context sentences")
    parser.add_argument("--num-prompts", type=int, default=300, help="Number of prompts to generate")

    # Quality
    parser.add_argument("--min-score", type=float, default=7.0, help="Minimum average quality score to keep")
    parser.add_argument("--skip-grading", action="store_true", help="Skip quality grading (keep all)")

    # Output
    parser.add_argument("--output-dir", default="./distillation_data", help="Output directory")
    parser.add_argument("--train-split", type=float, default=0.8)
    parser.add_argument("--val-split", type=float, default=0.1)

    args = parser.parse_args()

    client, method = get_clients(
        base_url=args.base_url, azure_endpoint=args.endpoint,
        project_endpoint=args.project_endpoint, api_key=args.api_key
    )
    judge = args.judge or args.teacher

    # Step 0: Verify deployments exist
    print(f"Verifying deployment '{args.teacher}'...")
    if not verify_deployment(client, args.teacher):
        print(f"  ERROR: Deployment '{args.teacher}' not found. Available deployments can be listed in Azure Portal.")
        sys.exit(1)
    print(f"  ✅ Teacher deployment verified.")

    if judge != args.teacher:
        print(f"Verifying judge deployment '{judge}'...")
        if not verify_deployment(client, judge):
            print(f"  ERROR: Judge deployment '{judge}' not found.")
            sys.exit(1)
        print(f"  ✅ Judge deployment verified.")

    # Step 1: Generate or load prompts
    if args.prompts_file:
        with open(args.prompts_file, encoding="utf-8") as pf:
            prompts = [line.strip() for line in pf if line.strip()]
        print(f"Loaded {len(prompts)} prompts from {args.prompts_file}")
    else:
        topics = [t.strip() for t in (args.topics or "general knowledge").split(",")]
        formats = [f.strip() for f in args.formats.split(",")]
        contexts = [c.strip() for c in args.contexts.split(",") if c.strip()] or [""]
        prompts = generate_combinatorial_prompts(topics, formats, contexts, args.num_prompts)
        print(f"Generated {len(prompts)} prompts ({len(topics)} topics × {len(formats)} formats × {len(contexts)} contexts)")

    # Step 2: Teacher generates responses
    print(f"\nTeacher ({args.teacher}) generating responses...")
    examples = []
    for i, prompt in enumerate(prompts):
        response = teacher_generate(client, args.teacher, args.system_prompt, prompt)
        if response:
            examples.append({"prompt": prompt, "response": response})
        if (i + 1) % 25 == 0:
            print(f"  {i+1}/{len(prompts)} ({len(examples)} successful)")
    print(f"  Teacher produced {len(examples)}/{len(prompts)} responses")

    # Step 3: Quality grade and filter
    if not args.skip_grading:
        print(f"\nGrading with {judge}...")
        for i, ex in enumerate(examples):
            scores = grade_output(client, judge, ex["response"])
            if scores:
                ex["scores"] = scores
                ex["avg_score"] = sum(scores.values()) / len(scores)
            else:
                ex["avg_score"] = 0
            if (i + 1) % 25 == 0:
                print(f"  Graded {i+1}/{len(examples)}")

        filtered = [ex for ex in examples if ex["avg_score"] >= args.min_score]
        avgs = [ex["avg_score"] for ex in examples if ex["avg_score"] > 0]
        print(f"  Passed filter (>= {args.min_score}): {len(filtered)}/{len(examples)}")
        if avgs:
            print(f"  Scores: min={min(avgs):.1f}, max={max(avgs):.1f}, mean={sum(avgs)/len(avgs):.1f}")
    else:
        filtered = examples
        print(f"Skipping grading — keeping all {len(filtered)} examples")

    # Step 4: Convert to SFT format and split
    sft_data = [{"messages": [
        {"role": "system", "content": args.system_prompt},
        {"role": "user", "content": ex["prompt"]},
        {"role": "assistant", "content": ex["response"]},
    ]} for ex in filtered]

    random.shuffle(sft_data)
    n = len(sft_data)
    t_end = int(n * args.train_split)
    v_end = int(n * (args.train_split + args.val_split))
    splits = {"train": sft_data[:t_end], "validation": sft_data[t_end:v_end], "test": sft_data[v_end:]}

    os.makedirs(args.output_dir, exist_ok=True)
    for name, data in splits.items():
        path = os.path.join(args.output_dir, f"{name}.jsonl")
        with open(path, "w", encoding="utf-8") as f:
            for ex in data:
                f.write(json.dumps(ex, ensure_ascii=False) + "\n")
        print(f"  {name}: {len(data)} examples → {path}")

    print(f"\n✅ Done! Dataset ready in {args.output_dir}/")


if __name__ == "__main__":
    main()

# /// script
# dependencies = [
#   "openai>=1.0",
#   "azure-identity",
#   "azure-ai-projects",
# ]
# ///
"""
monitor_training.py — Monitor a fine-tuning job until completion.

Polls the job status and streams training events (reward, loss, errors)
in real time. Exits when the job reaches a terminal state.

Usage:
  python monitor_training.py --job-id ftjob-abc123
  python monitor_training.py --base-url https://<resource>.services.ai.azure.com/api/projects/<project>/openai/v1/ --api-key KEY --job-id ftjob-abc123
  python monitor_training.py --job-id ftjob-abc123 --poll-interval 30
"""

import argparse
import os
import sys

try:
    sys.stdout.reconfigure(encoding="utf-8")
    sys.stderr.reconfigure(encoding="utf-8")
except (AttributeError, OSError):
    pass  # Stream not reconfigurable (older Python or non-tty); default encoding is fine
import time

sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from common import HelpOnErrorParser, get_clients

TERMINAL_STATUSES = {"succeeded", "failed", "cancelled"}


def monitor_job(client, job_id, poll_interval=15):
    """Poll a fine-tuning job until it reaches a terminal state."""
    # Cap memory for long-running jobs (RFT can run hours/days, accumulating thousands of events)
    seen_events = set()
    MAX_SEEN_EVENTS = 5000

    print(f"Monitoring job: {job_id}")
    print(f"Polling every {poll_interval}s. Ctrl+C to stop.\n")

    while True:
        try:
            job = client.fine_tuning.jobs.retrieve(job_id)
        except Exception as e:
            print(f"⚠️ Error retrieving job: {e}")
            time.sleep(poll_interval)
            continue

        status = (job.status or "").lower()

        # Fetch and display new events
        try:
            events = list(client.fine_tuning.jobs.list_events(job_id, limit=20))
            for event in reversed(events):
                event_key = (event.created_at, event.message)
                if event_key not in seen_events:
                    if len(seen_events) >= MAX_SEEN_EVENTS:
                        # Keep only the most recent half — a fully-flushed dedup window
                        # would risk re-printing old events on transient API hiccups, but
                        # without trimming this set grows unbounded for long RFT runs.
                        seen_events = set(list(seen_events)[-(MAX_SEEN_EVENTS // 2):])
                    seen_events.add(event_key)
                    ts = time.strftime("%H:%M:%S", time.localtime(event.created_at))
                    level = event.level or "info"

                    # Highlight step events
                    if "Step" in event.message and "reward" in event.message:
                        print(f"  📈 [{ts}] {event.message}")
                    elif "Step" in event.message and "loss" in event.message:
                        print(f"  📉 [{ts}] {event.message}")
                    elif "error" in event.message.lower() or level == "error":
                        print(f"  ❌ [{ts}] {event.message}")
                    elif "started" in event.message.lower() or "completed" in event.message.lower():
                        print(f"  🔔 [{ts}] {event.message}")
                    else:
                        print(f"  ℹ️ [{ts}] {event.message}")
        except Exception:
            pass  # Events API may not be available for all job states

        # Check terminal state
        if status in TERMINAL_STATUSES:
            print(f"\n{'='*50}")
            if status == "succeeded":
                model = job.fine_tuned_model or "unknown"
                print(f"  ✅ Job succeeded!")
                print(f"  Fine-tuned model: {model}")
                if job.trained_tokens:
                    print(f"  Trained tokens: {job.trained_tokens:,}")
            elif status == "failed":
                print(f"  ❌ Job failed.")
                if hasattr(job, "error") and job.error:
                    print(f"  Error: {job.error}")
            elif status == "cancelled":
                print(f"  ⚠️ Job was cancelled.")
            print(f"{'='*50}")
            return status

        time.sleep(poll_interval)


def build_parser():
    parser = HelpOnErrorParser(
        description="Monitor a fine-tuning job until completion",
        epilog=(
            "Example:\n"
            "  python monitor_training.py --job-id ftjob-abc123\n"
            "  python monitor_training.py --base-url https://<resource>.services.ai.azure.com/api/projects/<project>/openai/v1/ --api-key KEY --job-id ftjob-abc123"
        ),
        formatter_class=argparse.RawTextHelpFormatter,
    )
    parser.add_argument("--base-url", default=os.environ.get("OPENAI_BASE_URL"), help="Project /v1/ endpoint URL")
    parser.add_argument("--endpoint", default=os.environ.get("AZURE_OPENAI_ENDPOINT"),
                        help="Azure OpenAI endpoint (fallback)")
    parser.add_argument("--api-key", default=os.environ.get("AZURE_OPENAI_API_KEY"), help="API key")
    parser.add_argument("--project-endpoint", default=os.environ.get("AZURE_AI_PROJECT_ENDPOINT"),
                        help="Azure AI project endpoint (alternative to --base-url)")
    parser.add_argument("--job-id", required=True, help="Fine-tuning job ID (e.g., ftjob-abc123)")
    parser.add_argument("--poll-interval", type=int, default=15, help="Seconds between status checks (default: 15)")
    return parser


if __name__ == "__main__":
    parser = build_parser()
    if len(sys.argv) == 1:
        parser.print_help()
        sys.exit(0)

    args = parser.parse_args()
    client, method = get_clients(base_url=args.base_url, azure_endpoint=args.endpoint, project_endpoint=args.project_endpoint, api_key=args.api_key)
    status = monitor_job(client, args.job_id, args.poll_interval)
    sys.exit(0 if status == "succeeded" else 1)

# /// script
# dependencies = [
#   "openai>=1.0",
#   "azure-identity",
# ]
# ///
"""
score_dataset.py — Assess training data quality using an LLM judge.

Scores each example on correctness and relevance, optionally filters
out low-quality examples.

Usage:
  # Score all examples
  python score_dataset.py --input training.jsonl --output scored.jsonl

  # Score and filter (keep only score >= 7)
  python score_dataset.py --input training.jsonl --output filtered.jsonl --min-score 7

  # Custom scoring dimensions
  python score_dataset.py --input training.jsonl --output scored.jsonl \
      --dimensions "correctness,clarity,completeness"
"""

import json
import os
import re
import sys

try:
    sys.stdout.reconfigure(encoding="utf-8")
    sys.stderr.reconfigure(encoding="utf-8")
except (AttributeError, OSError):
    pass  # Stream not reconfigurable (older Python or non-tty); default encoding is fine
import time
from concurrent.futures import ThreadPoolExecutor, as_completed
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from common import HelpOnErrorParser, get_clients, _clamp_score


QUALITY_PROMPT = """You are a data quality assessor for machine learning training data.

## Task
Evaluate this training example for quality.

## User input (what the model receives)
{user_content}

## Assistant output (what the model should learn to produce)
{assistant_content}

## Scoring dimensions
{dimensions_text}

Rate each dimension on a scale of 1-10.

Return ONLY a JSON object with dimension names as keys and integer scores as values.
Example: {example_json}"""


DEFAULT_DIMENSIONS = {
    "correctness": "Is the assistant's output factually/functionally correct?",
    "relevance": "Does the output directly address the user's request?",
    "quality": "Is the output well-written, well-formatted, and professional?",
}


def score_example(client, model, user_content, assistant_content, dimensions):
    """Score a single training example."""
    dims_text = "\n".join(f"**{k}** (1-10): {v}" for k, v in dimensions.items())
    example = {k: 8 for k in dimensions}

    prompt = QUALITY_PROMPT.format(
        user_content=user_content[:2000],
        assistant_content=assistant_content[:2000],
        dimensions_text=dims_text,
        example_json=json.dumps(example),
    )

    for attempt in range(3):
        try:
            resp = client.chat.completions.create(
                model=model,
                messages=[{"role": "user", "content": prompt}],
                temperature=0.0,
                max_completion_tokens=200,
            )
            text = (resp.choices[0].message.content or "").strip()
            match = re.search(r'\{[^}]+\}', text)
            if match:
                scores = json.loads(match.group())
                return {k: _clamp_score(scores.get(k)) for k in dimensions}
        except Exception:
            if attempt < 2:
                time.sleep(2)

    return {k: 0 for k in dimensions}


def main():
    parser = HelpOnErrorParser(description="Score training data quality with LLM judge")
    parser.add_argument("--base-url", default=os.environ.get("OPENAI_BASE_URL"),
                        help="Project /v1/ URL (preferred)")
    parser.add_argument("--endpoint", default=os.environ.get("AZURE_OPENAI_ENDPOINT"),
                        help="Azure OpenAI endpoint (fallback)")
    parser.add_argument("--project-endpoint", default=os.environ.get("AZURE_AI_PROJECT_ENDPOINT"),
                        help="Azure AI project endpoint (Foundry SDK)")
    parser.add_argument("--api-key", default=os.environ.get("AZURE_OPENAI_API_KEY"))
    parser.add_argument("--model", default="gpt-4o", help="Judge model")
    parser.add_argument("--input", required=True, help="Input JSONL file")
    parser.add_argument("--output", required=True, help="Output JSONL file (with scores)")
    parser.add_argument("--min-score", type=float, default=None,
                        help="Minimum average score to keep (filters below this)")
    parser.add_argument("--dimensions", default=None,
                        help="Comma-separated dimension names (default: correctness,relevance,quality)")
    parser.add_argument("--concurrency", type=int, default=4, help="Parallel scoring workers")
    parser.add_argument("--strip-metadata", action="store_true",
                        help="Remove _quality_scores and _avg_quality from output (safe for training input)")
    args = parser.parse_args()

    client, method = get_clients(
        base_url=args.base_url, azure_endpoint=args.endpoint,
        project_endpoint=args.project_endpoint, api_key=args.api_key
    )

    # Parse dimensions
    if args.dimensions:
        dim_names = [d.strip() for d in args.dimensions.split(",")]
        dimensions = {d: f"Rate the {d} of the output" for d in dim_names}
    else:
        dimensions = DEFAULT_DIMENSIONS

    # Load data
    examples = []
    with open(args.input, encoding="utf-8") as f:
        for i, line in enumerate(f):
            if not line.strip():
                continue
            try:
                ex = json.loads(line)
            except json.JSONDecodeError as e:
                print(f"⚠️ Skipping malformed JSON on line {i+1}: {e}")
                continue
            msgs = ex.get("messages", [])
            user = next((m["content"] for m in msgs if m["role"] == "user"), "")
            asst = next((m["content"] for m in msgs if m["role"] == "assistant"), "")
            examples.append({"data": ex, "user": user, "assistant": asst})

    print(f"Loaded {len(examples)} examples. Scoring with {args.model}...")

    # Score in parallel
    def score_one(idx):
        ex = examples[idx]
        scores = score_example(client, args.model, ex["user"], ex["assistant"], dimensions)
        return idx, scores

    with ThreadPoolExecutor(max_workers=args.concurrency) as pool:
        futures = {pool.submit(score_one, i): i for i in range(len(examples))}
        done = 0
        for future in as_completed(futures):
            idx, scores = future.result()
            examples[idx]["scores"] = scores
            done += 1
            if done % 25 == 0:
                print(f"  Scored {done}/{len(examples)}")

    # Calculate stats
    all_avgs = []
    for ex in examples:
        scores = ex.get("scores", {})
        if scores and any(v > 0 for v in scores.values()):
            avg = sum(scores.values()) / len(scores)
            ex["avg_score"] = avg
            all_avgs.append(avg)

    if all_avgs:
        print(f"\nQuality Distribution:")
        print(f"  Mean:   {sum(all_avgs)/len(all_avgs):.1f}")
        print(f"  Min:    {min(all_avgs):.1f}")
        print(f"  Max:    {max(all_avgs):.1f}")
        sorted_avgs = sorted(all_avgs)
        n_avgs = len(sorted_avgs)
        if n_avgs % 2 == 1:
            median = sorted_avgs[n_avgs // 2]
        else:
            median = (sorted_avgs[n_avgs // 2 - 1] + sorted_avgs[n_avgs // 2]) / 2
        print(f"  Median: {median:.1f}")

    # Filter and write
    kept = 0
    filtered = 0
    with open(args.output, "w", encoding="utf-8") as f:
        for ex in examples:
            if not args.strip_metadata:
                ex["data"]["_quality_scores"] = ex.get("scores", {})
                ex["data"]["_avg_quality"] = ex.get("avg_score", 0)

            if args.min_score and ex.get("avg_score", 0) < args.min_score:
                filtered += 1
                continue

            f.write(json.dumps(ex["data"], ensure_ascii=False) + "\n")
            kept += 1

    print(f"\nKept: {kept}, Filtered: {filtered}")
    if args.min_score:
        print(f"(min_score threshold: {args.min_score})")
    if args.strip_metadata:
        print("(metadata stripped — output is safe for training input)")
    print(f"Output: {args.output}")


if __name__ == "__main__":
    main()

# /// script
# dependencies = [
#   "openai>=1.0",
#   "requests",
#   "azure-identity",
#   "azure-ai-projects",
# ]
# ///
"""
submit_training.py — Submit SFT, DPO, or RFT training jobs on Azure AI Foundry.

Handles both SDK and REST API submission (REST fallback for OSS models).
Supports /v1/ project endpoint (preferred) and Azure endpoint (fallback).

Usage:
  python submit_training.py --base-url https://<resource>.services.ai.azure.com/api/projects/<project>/openai/v1/ \
      --api-key KEY --training-file training.jsonl --validation-file validation.jsonl \
      --model gpt-4.1-mini --type sft --epochs 2 --lr 1.0

  python submit_training.py --endpoint https://<resource>.openai.azure.com --api-key KEY \
      --training-file-id file-abc123 --validation-file-id file-def456 \
      --model gpt-oss-20b --type sft --epochs 2 --lr 0.5 --use-rest

  python submit_training.py --base-url <url> --api-key KEY \
      --training-file-id file-abc123 --validation-file-id file-def456 \
      --model o4-mini-2025-04-16 --type rft --grader-file grader.py
"""

import json
import os
import sys


try:
    sys.stdout.reconfigure(encoding="utf-8")
    sys.stderr.reconfigure(encoding="utf-8")
except (AttributeError, OSError):
    pass  # Stream not reconfigurable (older Python or non-tty); default encoding is fine
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from common import HelpOnErrorParser, get_clients, upload_file

import requests


def submit_sft_sdk(client, model, train_id, val_id, epochs=2, lr=1.0, batch_size=None, suffix=None, training_type="globalStandard"):
    """Submit SFT job using the Python SDK."""
    hp = {"n_epochs": epochs, "learning_rate_multiplier": lr}
    if batch_size:
        hp["batch_size"] = batch_size

    kwargs = dict(
        model=model,
        training_file=train_id,
        validation_file=val_id,
        method={"type": "supervised"},
        hyperparameters=hp,
        # Azure-specific: passed via extra_body since the OpenAI SDK has no
        # top-level trainingType kwarg.
        extra_body={"trainingType": training_type},
    )
    if suffix:
        kwargs["suffix"] = suffix

    job = client.fine_tuning.jobs.create(**kwargs)
    return {"id": job.id, "status": job.status, "model": model, "method": "sdk"}


def submit_sft_rest(endpoint, api_key, model, train_id, val_id, epochs=2, lr=1.0, batch_size=None, suffix=None, training_type="globalStandard"):
    """Submit SFT job via REST API (fallback for models like gpt-oss-20b)."""
    url = f"{endpoint}/openai/fine_tuning/jobs?api-version=2025-04-01-preview"
    body = {
        "model": model,
        "training_file": train_id,
        "validation_file": val_id,
        "method": {"type": "supervised"},
        "hyperparameters": {"n_epochs": epochs, "learning_rate_multiplier": lr},
        "trainingType": training_type,
    }
    if batch_size:
        body["hyperparameters"]["batch_size"] = batch_size
    if suffix:
        body["suffix"] = suffix

    resp = requests.post(url, headers={
        "Content-Type": "application/json",
        "api-key": api_key,
    }, json=body, timeout=(10, 60))

    if resp.status_code in (200, 201):
        try:
            data = resp.json()
        except ValueError:
            raise RuntimeError(
                f"REST submission returned {resp.status_code} but body was not JSON: {resp.text[:200]}"
            )
        if "id" not in data or "status" not in data:
            raise RuntimeError(f"REST response missing 'id' or 'status' fields: {data}")
        return {"id": data["id"], "status": data["status"], "model": model, "method": "rest"}
    else:
        try:
            err_msg = resp.json().get('error', {}).get('message', 'Unknown error')
        except (ValueError, KeyError):
            err_msg = resp.text[:200] if resp.text else "Unknown error"
        raise RuntimeError(
            f"REST submission failed ({resp.status_code}): {err_msg}"
        )


def submit_rft(client, model, train_id, val_id, grader_source):
    """Submit RFT job."""
    job = client.fine_tuning.jobs.create(
        model=model,
        training_file=train_id,
        validation_file=val_id,
        method={
            "type": "reinforcement",
            "reinforcement": {
                "grader": {
                    "type": "python",
                    "name": "custom_grader",
                    "source": grader_source,
                },
            },
        },
    )
    return {"id": job.id, "status": job.status, "model": model, "method": "sdk-rft"}


def submit_dpo(client, model, train_id, val_id, epochs=2, lr=1.0, beta=0.1, suffix=None):
    """Submit DPO job."""
    job = client.fine_tuning.jobs.create(
        model=model,
        training_file=train_id,
        validation_file=val_id,
        suffix=suffix or None,
        method={
            "type": "dpo",
            "dpo": {
                "hyperparameters": {
                    "n_epochs": epochs,
                    "beta": beta,
                    "learning_rate_multiplier": lr,
                },
            },
        },
    )
    return {"id": job.id, "status": job.status, "model": model, "method": "sdk-dpo"}


def main():
    parser = HelpOnErrorParser(description="Submit fine-tuning jobs on Azure AI Foundry")
    parser.add_argument("--base-url", default=os.environ.get("OPENAI_BASE_URL"),
                        help="Project /v1/ URL (preferred)")
    parser.add_argument("--endpoint", default=os.environ.get("AZURE_OPENAI_ENDPOINT"),
                        help="Azure OpenAI endpoint (fallback)")
    parser.add_argument("--project-endpoint", default=os.environ.get("AZURE_AI_PROJECT_ENDPOINT"),
                        help="Azure AI project endpoint (Foundry SDK)")
    parser.add_argument("--api-key", default=os.environ.get("AZURE_OPENAI_API_KEY"),
                        help="API key")
    parser.add_argument("--model", required=True, help="Base model name (e.g., gpt-4.1-mini)")
    parser.add_argument("--type", choices=["sft", "dpo", "rft"], default="sft",
                        help="Training type: sft, dpo, or rft")

    # Data files — either paths (will upload) or IDs (already uploaded)
    parser.add_argument("--training-file", help="Path to training JSONL file (will upload)")
    parser.add_argument("--validation-file", help="Path to validation JSONL file (will upload)")
    parser.add_argument("--training-file-id", help="Already-uploaded training file ID")
    parser.add_argument("--validation-file-id", help="Already-uploaded validation file ID")

    # Hyperparameters
    parser.add_argument("--epochs", type=int, default=2)
    parser.add_argument("--lr", type=float, default=1.0, help="Learning rate multiplier")
    parser.add_argument("--batch-size", type=int, default=None)
    parser.add_argument("--suffix", help="Model suffix for identification")

    # DPO-specific
    parser.add_argument("--beta", type=float, default=0.1, help="DPO beta (alignment strength)")

    # RFT-specific
    parser.add_argument("--grader-file", help="Path to Python grader file (for RFT)")

    # REST fallback
    parser.add_argument("--use-rest", action="store_true",
                        help="Force REST API (needed for gpt-oss-20b and other OSS models)")
    parser.add_argument("--training-type", choices=["globalStandard", "developerTier", "standard"],
                        default="globalStandard",
                        help="Azure training tier (default: globalStandard). developerTier is ~50%% off "
                             "globalStandard with lower quotas. OSS models (gpt-oss-20b, Ministral, "
                             "Llama, Qwen) only support globalStandard.")

    args = parser.parse_args()

    client, method = get_clients(
        base_url=args.base_url, azure_endpoint=args.endpoint,
        project_endpoint=args.project_endpoint, api_key=args.api_key
    )

    # Resolve file IDs
    train_id = args.training_file_id
    val_id = args.validation_file_id
    if args.training_file:
        train_id = upload_file(client, args.training_file)
    if args.validation_file:
        val_id = upload_file(client, args.validation_file)

    if not train_id or not val_id:
        print("Error: Provide training and validation file paths or IDs")
        sys.exit(1)

    # Submit
    if args.type == "rft":
        if not args.grader_file:
            print("Error: --grader-file required for RFT")
            sys.exit(1)
        with open(args.grader_file, encoding="utf-8") as f:
            grader_source = f.read()
        result = submit_rft(client, args.model, train_id, val_id, grader_source)
    elif args.type == "dpo":
        result = submit_dpo(client, args.model, train_id, val_id,
                            args.epochs, args.lr, args.beta, args.suffix)
    elif args.use_rest:
        if not args.endpoint or not args.api_key:
            print("Error: --use-rest requires --endpoint and --api-key (REST does not support DefaultAzureCredential)")
            sys.exit(1)
        result = submit_sft_rest(args.endpoint, args.api_key, args.model,
                                 train_id, val_id, args.epochs, args.lr, args.batch_size, args.suffix,
                                 args.training_type)
    else:
        # SFT via SDK with REST fallback for OSS models
        try:
            result = submit_sft_sdk(client, args.model, train_id, val_id,
                                    args.epochs, args.lr, args.batch_size, args.suffix,
                                    args.training_type)
        except Exception as e:
            err_str = str(e).lower()
            # Match a wider set of "use REST instead" signals than the original
            # exact-string comparison: Azure changes error text periodically.
            if ("trainingtype" in err_str
                    or "globalstandard" in err_str
                    or "global_standard" in err_str
                    or "does not support fine-tuning" in err_str):
                if not args.endpoint or not args.api_key:
                    print(f"SDK failed for {args.model}. REST fallback requires --endpoint and --api-key.")
                    sys.exit(1)
                print(f"SDK failed for {args.model}, falling back to REST API...")
                result = submit_sft_rest(args.endpoint, args.api_key, args.model,
                                         train_id, val_id, args.epochs, args.lr, args.batch_size, args.suffix,
                                         args.training_type)
            else:
                raise

    print(f"\nJob submitted successfully:")
    print(json.dumps(result, indent=2))

    # Save job info
    outfile = f"ft_job_{result['id']}.json"
    with open(outfile, "w", encoding="utf-8") as f:
        json.dump({**result, "epochs": args.epochs, "lr": args.lr,
                    "batch_size": args.batch_size, "train_file": train_id,
                    "val_file": val_id}, f, indent=2)
    print(f"Job info saved to {outfile}")


if __name__ == "__main__":
    main()

# Validation scripts for SFT, DPO, and RFT JSONL files.
# Usage:
#   python -m scripts.validate.validate_sft <path-to-jsonl>
#   python -m scripts.validate.validate_dpo <path-to-jsonl>
#   python -m scripts.validate.validate_rft <path-to-jsonl>
#   python -m scripts.validate.data_stats   <path-to-jsonl>

#!/usr/bin/env python3
"""Compute dataset statistics for any fine-tuning JSONL file.

Adapted from foundry-ft agent. Auto-detects SFT/DPO/RFT format and reports
token estimates, role distribution, and rough cost estimates.
"""
import json
import sys

try:
    sys.stdout.reconfigure(encoding="utf-8")
    sys.stderr.reconfigure(encoding="utf-8")
except (AttributeError, OSError):
    pass  # Stream not reconfigurable (older Python or non-tty); default encoding is fine
from collections import Counter


def estimate_tokens(text: str) -> int:
    """Rough token estimate: ~4 chars per token for English text."""
    return max(1, len(text) // 4)


def extract_text(record: dict) -> str:
    """Extract all text content from a record regardless of format."""
    texts = []
    if "messages" in record:
        for msg in record["messages"]:
            if "content" in msg and msg["content"]:
                texts.append(str(msg["content"]))
    if "input" in record and "messages" in record["input"]:
        for msg in record["input"]["messages"]:
            if "content" in msg and msg["content"]:
                texts.append(str(msg["content"]))
    for field in ["preferred_output", "non_preferred_output"]:
        if field in record:
            for msg in record[field]:
                if "content" in msg and msg["content"]:
                    texts.append(str(msg["content"]))
    # Include any extra fields beyond messages/input/preferred_output/non_preferred_output
    known_structural = {"messages", "input", "preferred_output", "non_preferred_output"}
    for field in record:
        if field not in known_structural and isinstance(record[field], (str, int, float)):
            texts.append(str(record[field]))
    return " ".join(texts)


def data_stats(filepath: str) -> None:
    records = []
    format_type = "unknown"
    parse_errors = 0

    with open(filepath, "r", encoding="utf-8") as f:
        for line in f:
            line = line.strip()
            if not line:
                continue
            try:
                records.append(json.loads(line))
            except json.JSONDecodeError:
                parse_errors += 1

    if not records:
        print(f"No valid records found in {filepath}")
        sys.exit(1)

    # Detect format
    first = records[0]
    if "input" in first and "preferred_output" in first:
        format_type = "DPO"
    elif "messages" in first:
        msgs = first["messages"]
        extra_fields = set(first.keys()) - {"messages"}
        last_role = msgs[-1].get("role") if isinstance(msgs, list) and msgs else None
        if extra_fields and last_role == "user":
            format_type = "RFT"
        else:
            format_type = "SFT"

    # Compute stats
    token_counts = [estimate_tokens(extract_text(r)) for r in records]
    total_tokens = sum(token_counts)
    avg_tokens = total_tokens / len(records)
    min_tokens = min(token_counts)
    max_tokens = max(token_counts)

    print(f"\n{'='*60}")
    print(f"Dataset Statistics: {filepath}")
    print(f"{'='*60}")
    print(f"Format:           {format_type}")
    print(f"Total records:    {len(records)}")
    print(f"Parse errors:     {parse_errors}")
    print(f"")
    print(f"Token Estimates (approx):")
    print(f"  Total:          {total_tokens:,}")
    print(f"  Average/record: {avg_tokens:,.0f}")
    print(f"  Min:            {min_tokens:,}")
    print(f"  Max:            {max_tokens:,}")

    if format_type == "SFT":
        role_counts = Counter()
        for r in records:
            for msg in r.get("messages", []):
                role_counts[msg.get("role", "unknown")] += 1
        print(f"\nRole Distribution:")
        for role, count in role_counts.most_common():
            print(f"  {role}: {count}")

        has_system = sum(1 for r in records if any(m.get("role") == "system" for m in r.get("messages", [])))
        print(f"\nRecords with system message: {has_system}/{len(records)}")

    elif format_type == "DPO":
        pref_lens = []
        non_pref_lens = []
        for r in records:
            pref_text = " ".join(m.get("content", "") for m in r.get("preferred_output", []))
            non_pref_text = " ".join(m.get("content", "") for m in r.get("non_preferred_output", []))
            pref_lens.append(estimate_tokens(pref_text))
            non_pref_lens.append(estimate_tokens(non_pref_text))
        print(f"\nPreferred output avg tokens:     {sum(pref_lens)/len(pref_lens):,.0f}")
        print(f"Non-preferred output avg tokens: {sum(non_pref_lens)/len(non_pref_lens):,.0f}")

    elif format_type == "RFT":
        grader_field_counts = Counter()
        grader_values = []
        for r in records:
            extra = set(r.keys()) - {"messages"}
            grader_field_counts.update(extra)
            for field in sorted(extra):
                grader_values.append(str(r[field]))
        unique = len(set(grader_values))
        avg_val_len = sum(len(v) for v in grader_values) / len(grader_values) if grader_values else 0
        print(f"\nGrader fields found:")
        for field, count in grader_field_counts.most_common():
            print(f"  • '{field}' — in {count}/{len(records)} records")
        print(f"Unique grader values: {unique}/{len(grader_values)}")
        print(f"Avg grader value length: {avg_val_len:.0f} chars")

    # Dataset size guidance
    print(f"\n📊 Dataset size guidance:")
    if len(records) < 50:
        print(f"  ⚠️ Very small dataset ({len(records)} records). May only learn format, not domain knowledge.")
    elif len(records) < 200:
        print(f"  ⚠️ Small dataset. Good for initial experiments — evaluate results and add more data if needed.")
    elif len(records) <= 500:
        print(f"  ✅ Sweet spot for getting started (200-500). Evaluate results to decide if you need more.")
    elif len(records) <= 2000:
        print(f"  ✅ Good dataset size. Watch for diminishing returns — check if quality beats quantity.")
    else:
        print(f"  ⚠️ Large dataset ({len(records):,}). Larger isn't always better — especially for OSS models where 335-500 examples outperformed 4K.")


if __name__ == "__main__":
    if len(sys.argv) != 2:
        print("Usage: python data_stats.py <path-to-jsonl>")
        sys.exit(1)
    data_stats(sys.argv[1])

#!/usr/bin/env python3
"""Validate DPO (Direct Preference Optimization) JSONL files for Azure AI Foundry.

Adapted from foundry-ft agent with additional checks:
- Identical preferred/non_preferred detection
- DPO overtraining risk (small dataset warning)
"""
import json
import sys



try:
    sys.stdout.reconfigure(encoding="utf-8")
    sys.stderr.reconfigure(encoding="utf-8")
except (AttributeError, OSError):
    pass  # Stream not reconfigurable (older Python or non-tty); default encoding is fine
def validate_dpo(filepath: str) -> None:
    errors = []
    warnings = []
    total = 0

    with open(filepath, "r", encoding="utf-8") as f:
        for line_num, line in enumerate(f, 1):
            line = line.strip()
            if not line:
                continue
            total += 1

            try:
                record = json.loads(line)
            except json.JSONDecodeError as e:
                errors.append(f"Line {line_num}: Invalid JSON — {e}")
                continue

            for field in ["input", "preferred_output", "non_preferred_output"]:
                if field not in record:
                    errors.append(f"Line {line_num}: Missing '{field}' field")

            if "input" not in record:
                continue

            inp = record["input"]
            if "messages" not in inp:
                errors.append(f"Line {line_num}: 'input' missing 'messages' field")
            else:
                msgs = inp["messages"]
                if not any(m.get("role") == "user" for m in msgs):
                    errors.append(f"Line {line_num}: 'input.messages' has no 'user' message")

            for output_field in ["preferred_output", "non_preferred_output"]:
                if output_field in record:
                    out = record[output_field]
                    if not isinstance(out, list) or len(out) == 0:
                        errors.append(f"Line {line_num}: '{output_field}' must be a non-empty array")
                    elif not any(m.get("role") == "assistant" for m in out):
                        errors.append(f"Line {line_num}: '{output_field}' has no 'assistant' message")

            if "preferred_output" in record and "non_preferred_output" in record:
                pref = json.dumps(record["preferred_output"], sort_keys=True)
                non_pref = json.dumps(record["non_preferred_output"], sort_keys=True)
                if pref == non_pref:
                    warnings.append(f"Line {line_num}: preferred and non_preferred outputs are identical")

    print(f"\n{'='*60}")
    print(f"DPO Validation Report: {filepath}")
    print(f"{'='*60}")
    print(f"Total records: {total}")
    print(f"Errors: {len(errors)}")
    print(f"Warnings: {len(warnings)}")

    # DPO-specific guidance from our experiments
    if total < 500 and total > 0:
        print(f"\n⚠️  DPO tip: With {total} pairs, use n_epochs=1-2 max (Azure defaults to 3, which causes overtraining on small datasets).")
    if total > 0:
        print(f"\n💡 DPO tip: If your base model already scores >9/10 on this task, DPO may hurt more than help.")

    if errors:
        print(f"\n❌ ERRORS (must fix):")
        for e in errors[:20]:
            print(f"  • {e}")
        if len(errors) > 20:
            print(f"  ... and {len(errors) - 20} more errors")

    if warnings:
        print(f"\n⚠️  WARNINGS:")
        for w in warnings[:10]:
            print(f"  • {w}")

    if not errors:
        print(f"\n✅ Data is valid for DPO fine-tuning!")
    else:
        print(f"\n❌ Fix {len(errors)} error(s) before submitting.")
        sys.exit(1)


if __name__ == "__main__":
    if len(sys.argv) != 2:
        print("Usage: python validate_dpo.py <path-to-jsonl>")
        sys.exit(1)
    validate_dpo(sys.argv[1])

#!/usr/bin/env python3
"""Validate RFT (Reinforcement Fine-Tuning) JSONL files for Azure AI Foundry.

Adapted from foundry-ft agent with critical additions from our platform gotchas:
- Grader escaping warnings for newlines (\\n must be \\\\n in JSON strings)
- Content moderation risk detection ("chain of thought" triggers RAI filter)
- Reference answer diversity check
"""
import argparse
import json
import sys

try:
    sys.stdout.reconfigure(encoding="utf-8")
    sys.stderr.reconfigure(encoding="utf-8")
except (AttributeError, OSError):
    pass  # Stream not reconfigurable (older Python or non-tty); default encoding is fine
from collections import Counter


RISKY_PHRASES = [
    "chain of thought", "step by step reasoning", "let me think",
    "think carefully", "reason through",
]


def validate_rft(filepath, expected_field=None):
    errors = []
    warnings = []
    total = 0
    extra_fields_per_line: list[set[str]] = []
    all_extra_field_counts: Counter = Counter()
    grader_values: list[str] = []

    with open(filepath, "r", encoding="utf-8") as f:
        for line_num, line in enumerate(f, 1):
            raw_line = line
            line = line.strip()
            if not line:
                continue
            total += 1

            try:
                record = json.loads(line)
            except json.JSONDecodeError as e:
                errors.append(f"Line {line_num}: Invalid JSON — {e}")
                continue

            if "messages" not in record:
                errors.append(f"Line {line_num}: Missing 'messages' field")
            else:
                msgs = record["messages"]
                if not isinstance(msgs, list) or len(msgs) == 0:
                    errors.append(f"Line {line_num}: 'messages' must be a non-empty array")
                elif not any(m.get("role") == "user" for m in msgs):
                    errors.append(f"Line {line_num}: 'messages' has no 'user' message")
                elif msgs[-1].get("role") != "user":
                    errors.append(
                        f"Line {line_num}: Last message must be 'user' role for RFT "
                        f"(found '{msgs[-1].get('role')}') — unlike SFT, the model generates its own response"
                    )

            # Detect extra fields (grader fields) beyond 'messages'
            extra_fields = set(record.keys()) - {"messages"}
            extra_fields_per_line.append(extra_fields)
            all_extra_field_counts.update(extra_fields)

            if expected_field:
                if expected_field not in record:
                    errors.append(f"Line {line_num}: Missing expected field '{expected_field}'")
                else:
                    val = str(record[expected_field]).strip()
                    if not val:
                        errors.append(f"Line {line_num}: '{expected_field}' is empty")
                    else:
                        grader_values.append(val)
            else:
                if not extra_fields:
                    errors.append(
                        f"Line {line_num}: No grader fields found — RFT requires at least "
                        "one field beyond 'messages' (e.g. 'answer', 'reference_code')"
                    )
                else:
                    # Collect values from extra fields for diversity check
                    for field in sorted(extra_fields):
                        val = str(record[field]).strip()
                        if val:
                            grader_values.append(val)

                    # Check for unescaped newlines in extra fields (CRITICAL platform gotcha)
                    # Instead of regex-parsing the raw JSON line (which risks catastrophic
                    # backtracking), we compare the parsed value against the raw line to
                    # detect single-escaped \n that should be double-escaped \\n.
                    for field in extra_fields:
                        parsed_val = str(record.get(field, ""))
                        if "\n" in parsed_val:
                            # The parsed value contains actual newlines — check if the raw
                            # JSON has them properly double-escaped
                            field_needle = f'"{field}"'
                            if field_needle in raw_line:
                                field_start = raw_line.index(field_needle)
                                field_region = raw_line[field_start:field_start + 500]
                                # Single-escaped \n in raw JSON (not \\n) means the source
                                # code newlines aren't properly escaped for the platform
                                if "\\n" in field_region and "\\\\n" not in field_region:
                                    warnings.append(
                                        f"Line {line_num}: '{field}' contains \\n sequences — "
                                        "if this is grader source code embedded in JSON, "
                                        "ensure newlines are escaped as \\\\n."
                                    )

            # Content moderation risk
            all_text = json.dumps(record).lower()
            for phrase in RISKY_PHRASES:
                if phrase in all_text:
                    warnings.append(
                        f"Line {line_num}: Contains '{phrase}' — may trigger Azure content moderation filter."
                    )
                    break

    # Check for inconsistent extra-field schemas across examples
    field_sets = [fs for fs in extra_fields_per_line if fs]
    if len(field_sets) > 1:
        first_schema = field_sets[0]
        inconsistent_lines = [
            i + 1 for i, fs in enumerate(extra_fields_per_line)
            if fs and fs != first_schema
        ]
        if inconsistent_lines:
            warnings.append(
                f"Inconsistent grader fields across examples — "
                f"line 1 has {sorted(first_schema)}, but {len(inconsistent_lines)} "
                f"line(s) differ (e.g. line {inconsistent_lines[0]}). "
                "Ensure your grader handles all field variants."
            )

    # Diversity check
    if grader_values:
        unique_values = set(grader_values)
        if len(unique_values) == 1:
            warnings.append(
                f"All grader field values are identical ('{list(unique_values)[0][:50]}...') — "
                "grader may not learn effectively"
            )
        avg_len = sum(len(v) for v in grader_values) / len(grader_values)
        if avg_len > 500:
            warnings.append(
                f"Average grader field value length is {avg_len:.0f} chars — "
                "consider using a model_grader instead of string_check"
            )

    print(f"\n{'='*60}")
    print(f"RFT Validation Report: {filepath}")
    print(f"{'='*60}")
    print(f"Total records: {total}")
    print(f"Errors: {len(errors)}")
    print(f"Warnings: {len(warnings)}")

    if all_extra_field_counts:
        print(f"\nGrader fields found:")
        for field, count in all_extra_field_counts.most_common():
            print(f"  • '{field}' — in {count}/{total} records")

    if errors:
        print(f"\n❌ ERRORS (must fix):")
        for e in errors[:20]:
            print(f"  • {e}")
        if len(errors) > 20:
            print(f"  ... and {len(errors) - 20} more errors")

    if warnings:
        print(f"\n⚠️  WARNINGS:")
        for w in warnings[:10]:
            print(f"  • {w}")
        if len(warnings) > 10:
            print(f"  ... and {len(warnings) - 10} more warnings")

    # RFT-specific guidance
    if total > 0:
        print(f"\n💡 RFT tips:")
        print(f"  • Ensure your training grader matches your eval grader (alignment gotcha)")
        print(f"  • Start with reasoning_effort='medium', pass_threshold=0.5")
        print(f"  • RFT is primarily for o-series models (o4-mini). Check Azure docs for the latest supported model list.")

    if not errors:
        print(f"\n✅ Data is valid for RFT fine-tuning!")
    else:
        print(f"\n❌ Fix {len(errors)} error(s) before submitting.")
        sys.exit(1)


if __name__ == "__main__":
    parser = argparse.ArgumentParser(
        description="Validate RFT (Reinforcement Fine-Tuning) JSONL files for Azure AI Foundry."
    )
    parser.add_argument("filepath", help="Path to the JSONL file to validate")
    parser.add_argument(
        "--expected-field",
        default=None,
        help="Specific grader field name to require (e.g. 'answer'). "
             "If omitted, any extra field beyond 'messages' is accepted.",
    )
    args = parser.parse_args()
    validate_rft(args.filepath, expected_field=args.expected_field)

#!/usr/bin/env python3
"""Validate SFT (Supervised Fine-Tuning) JSONL files for Azure AI Foundry.

Adapted from foundry-ft agent with additional checks from our platform gotchas:
- Token length warnings (4096 limit varies by model)
- System prompt consistency check
"""
import json
import sys


try:
    sys.stdout.reconfigure(encoding="utf-8")
    sys.stderr.reconfigure(encoding="utf-8")
except (AttributeError, OSError):
    pass  # Stream not reconfigurable (older Python or non-tty); default encoding is fine
VALID_ROLES = {"system", "user", "assistant", "tool"}


def estimate_tokens(text: str) -> int:
    """Rough token estimate: ~4 chars per token for English text."""
    return max(1, len(text) // 4)


def validate_sft(filepath: str) -> None:
    errors = []
    warnings = []
    total = 0
    token_counts = []
    system_prompts = set()

    with open(filepath, "r", encoding="utf-8") as f:
        for line_num, line in enumerate(f, 1):
            line = line.strip()
            if not line:
                continue
            total += 1

            try:
                record = json.loads(line)
            except json.JSONDecodeError as e:
                errors.append(f"Line {line_num}: Invalid JSON — {e}")
                continue

            if "messages" not in record:
                errors.append(f"Line {line_num}: Missing 'messages' field")
                continue

            messages = record["messages"]
            if not isinstance(messages, list) or len(messages) == 0:
                errors.append(f"Line {line_num}: 'messages' must be a non-empty array")
                continue

            roles_found = set()
            total_text = ""
            for i, msg in enumerate(messages):
                if "role" not in msg:
                    errors.append(f"Line {line_num}, message {i}: Missing 'role'")
                elif msg["role"] not in VALID_ROLES:
                    errors.append(f"Line {line_num}, message {i}: Invalid role '{msg['role']}' (expected: {VALID_ROLES})")
                else:
                    roles_found.add(msg["role"])

                if "content" not in msg and "tool_calls" not in msg:
                    errors.append(f"Line {line_num}, message {i}: Missing 'content' (and no 'tool_calls')")
                elif "content" in msg and msg["content"] is not None:
                    content = str(msg["content"])
                    if not content.strip():
                        warnings.append(f"Line {line_num}, message {i}: Empty content string")
                    total_text += content

                    if msg.get("role") == "system":
                        system_prompts.add(content.strip()[:100])

            if "user" not in roles_found:
                errors.append(f"Line {line_num}: No 'user' message found")
            if "assistant" not in roles_found:
                errors.append(f"Line {line_num}: No 'assistant' message found")

            tokens = estimate_tokens(total_text)
            token_counts.append(tokens)
            if tokens > 4096:
                warnings.append(f"Line {line_num}: ~{tokens} tokens (exceeds 4096 limit for most models)")

    # Report
    print(f"\n{'='*60}")
    print(f"SFT Validation Report: {filepath}")
    print(f"{'='*60}")
    print(f"Total records: {total}")
    print(f"Errors: {len(errors)}")
    print(f"Warnings: {len(warnings)}")

    if token_counts:
        avg_tok = sum(token_counts) / len(token_counts)
        print(f"\nToken stats (approx):")
        print(f"  Avg: {avg_tok:.0f}  Min: {min(token_counts)}  Max: {max(token_counts)}")
        print(f"  Total: {sum(token_counts):,}")

    if len(system_prompts) > 1:
        warnings.append(f"Found {len(system_prompts)} different system prompts — ensure this is intentional")
    if system_prompts:
        print(f"\nSystem prompts: {len(system_prompts)} unique")

    if errors:
        print(f"\n❌ ERRORS (must fix):")
        for e in errors[:20]:
            print(f"  • {e}")
        if len(errors) > 20:
            print(f"  ... and {len(errors) - 20} more errors")

    if warnings:
        print(f"\n⚠️  WARNINGS:")
        for w in warnings[:10]:
            print(f"  • {w}")

    if not errors:
        print(f"\n✅ Data is valid for SFT fine-tuning!")
    else:
        print(f"\n❌ Fix {len(errors)} error(s) before submitting.")
        sys.exit(1)


if __name__ == "__main__":
    if len(sys.argv) != 2:
        print("Usage: python validate_sft.py <path-to-jsonl>")
        sys.exit(1)
    validate_sft(sys.argv[1])

# Dataset Creation Workflow

Three paths to training data (these combine well: curate seeds → augment → generate at scale):

> If you already have data, skip to validation: `python scripts/validate/validate_sft.py your_data.jsonl`

## Approach 1: Manual Curation

Write examples by hand, collect from production logs, or adapt existing datasets.

**When to use:**
- You have real-world examples (production logs, support tickets, labeled data)
- Your task requires domain expertise an LLM can't reliably generate
- You need a gold-standard evaluation set (always curate manually)

**Tips:**
- Start with 10-20 examples to establish quality standards and format consistency
- These seed examples also serve as the foundation of your evaluation test set
- For RFT, you only need prompts + expected answers — no model responses needed

## Approach 2: LLM Augmentation

Expand a small curated dataset through **rephrasing** — generating diverse variations while keeping the same expected answer. Especially useful for RFT.

**When to use:**
- Well-defined task with clear correct answers
- You can write quality examples but need more volume
- Diversity of phrasing matters more than diversity of scenarios

**Workflow:**
1. Write base examples with correct expected answers
2. For each, use an LLM to generate rephrasings varying tone, detail, and wording
3. Each rephrasing gets the same expected answer — only the phrasing changes
4. Validate the augmented dataset

**Rephrasing prompt:**
```
Generate N different phrasings of this request. Each should:
- Use different wording, tone, or level of detail
- Include the same key identifiers (order IDs, item names)
- Vary between formal, casual, frustrated, brief, and detailed styles
Return a JSON array of N strings.

Original: [your example]
```

A cheap model (gpt-4.1-mini) works well — no new ground truth needed, just phrasing diversity.

## Approach 3: Synthetic Generation

Generate training data from scratch using LLM prompts.


1. Define topic/scenario categories for diversity
2. Generate prompts from an LLM
3. Generate responses (or preferred/non-preferred pairs for DPO)
4. Grade quality with an LLM judge
5. Filter to a quality threshold
6. Split into train/validation/test sets
7. Write JSONL in the correct format (see `references/dataset-formats.md`)

## Quality Checklist

Before training, verify:

- [ ] **No duplicates**: Exact or near-duplicate examples waste budget
- [ ] **Balanced distribution**: Topics, difficulty, output lengths well-distributed
- [ ] **Consistent formatting**: All examples follow the same structure
- [ ] **Correct outputs**: Spot-check 20 random examples manually
- [ ] **Reasonable lengths**: No extremely short or extremely long outputs
- [ ] **Clean text**: No encoding errors, garbled text, or template artifacts

## Dataset Size vs. Quality

From experiments:
- **335 high-quality examples** (carefully curated) → best combined eval score (9.15)
- **1,576 examples** (broader but noisier) → higher correctness but lower conciseness (8.53)

**Takeaway**: A small, pristine dataset usually beats a large, noisy one. Quality filter aggressively.

# Diagnosing Poor Results

When your fine-tuned model performs worse than expected, work through this checklist top-down (most common causes first).

## Diagnostic Table

| # | Symptom | Likely Cause | Fix |
|---|---------|-------------|-----|
| 1 | Training loss → 0, validation loss rises | Overfitting | 1) Deploy earlier checkpoint. 2) Reduce epochs. 3) Lower LR. 4) Add more diverse data. Overfitting ratio > 1.5 is concerning. |
| 2 | High correctness, low conciseness (or reverse) | Dataset style mismatch | **Verbose**: Add concise examples, use "Be concise" system prompt, filter to shortest correct examples. **Terse**: Add detailed examples, increase dataset with quality-filtered data. |
| 3 | Model seems good on spot-check but auto-eval is low | Evaluation rubric issue | Manually grade 10 examples vs. LLM judge. Check: Is judge model strong enough? Is rubric clear? Do reference answers match desired output? |
| 4 | Garbage, empty outputs, or errors | Deployment/client bug | Check: wrong model format (→ HTTP 500), `AzureOpenAI` on project endpoint (→ "api-version not allowed"), low capacity (→ timeouts), wrong deployment name. Test with curl. |
| 5 | RFT model scores below base model | RFT-specific issue | See RFT section below. |

## RFT-Specific Diagnosis

| Signal | Meaning | Fix |
|--------|---------|-----|
| Train-val grader gap > 0.2 | Model gaming the grader | Use stricter/more deterministic grader (Python execution > LLM judge) |
| Grader too easy | High grader scores but bad outputs | Add multi-criteria grading (syntax + semantic) |
| Grader too noisy | Random signal, no learning | Use deterministic grader or increase val set size |
| All of the above fail | RFT may not suit this task | Switch back to SFT |

## Escalation Path

If nothing above helps:

1. **Try a different base model** — some fine-tune better for certain tasks
2. **Increase dataset 2x-5x** with synthetic data
3. **Simplify the task** — fine-tune for a narrower sub-task first
4. **Try prompt engineering instead** — sometimes a well-crafted system prompt beats fine-tuning
5. **Combine approaches** — prompt engineering + fine-tuning together

## Red Flags: Don't Fine-Tune

- Base model already scores > 9.0 (minimal headroom)
- Task changes frequently (constant retraining needed)
- < 50 examples and can't generate synthetic data
- "Correct" output is highly subjective

# Full Pipeline Workflow

End-to-end fine-tuning on Azure AI Foundry in 9 phases.

## Prerequisites

- Azure AI Foundry resource with fine-tuning enabled
- Python 3.10+ with `openai` and `requests`
- Azure CLI (`az`) authenticated
- A clear task definition: what should the model do differently after fine-tuning?

## Phase 1: Define the Task

Answer before touching data or models:

1. **What task?** (e.g., "translate natural language to Python code")
2. **What does good output look like?** Write 5 examples by hand.
3. **What does bad output look like?** Write 3 anti-examples.
4. **How will you measure success?** Define evaluation dimensions (see `references/grader-design.md`).
5. **Which base model?** Pick 1-3 candidates from the supported model list.

## Phase 2: Prepare the Dataset

### Option A: You Have Data
1. Convert to SFT JSONL format (see `references/dataset-formats.md`)
2. Split: 80% train, 10% validation, 10% held-out test
3. Remove or fix low-quality examples

### Option B: Synthetic Data
1. Generate using LLM prompts (see `workflows/dataset-creation.md`)
2. Convert to SFT JSONL with `scripts/convert_dataset.py`

### Option C: Hybrid (Seed + Synthetic)
1. Use existing data as seed, generate synthetic variations
2. Merge, deduplicate, and quality-filter

**Checkpoint**: You should have `training.jsonl`, `validation.jsonl`, and `test.jsonl` (never used for training).

## Phase 3: Establish Baselines

1. Deploy base model (or use existing deployment)
2. Record scores — this is your "zero" that every fine-tune must beat

## Phase 4: Choose Training Type

See `references/training-types.md` for the full decision framework.

| Condition | Training Type |
|-----------|--------------|
| Have input-output pairs | SFT |
| Can write a grading function | RFT (reasoning models only) |
| Need style alignment | DPO |

Most projects start with SFT. Move to RFT/DPO only if SFT isn't sufficient.

## Phase 5: Upload and Submit Training

Use `scripts/submit_training.py` or the API directly. See `references/hyperparameters.md` for starting HP values.

**Foundry CLI** alternative (no Python):
```bash
azd ai finetuning jobs submit -f ./fine-tune-job.yaml
```

## Phase 6: Monitor and Analyze

1. Wait for completion or use `scripts/monitor_training.py`
2. Analyze training curves with `scripts/check_training.py`
3. Read `references/training-curves.md` to interpret results
4. Check for overfitting — consider deploying an earlier checkpoint if detected

## Phase 7: Evaluate Fine-Tuned Model

1. Deploy fine-tuned model (see `references/deployment.md` for format/SKU)
2. Compare against baseline and previous experiments
3. Delete deployment after evaluation

## Phase 8: Iterate

Follow `workflows/iterative-training.md`:
- Adjust hyperparameters based on training curves
- Try different data subsets or augmentations
- Test different base models
- Track everything in your leaderboard

## Phase 9: Ship

When the model convincingly beats baseline:
1. Deploy with production-appropriate capacity
2. Monitor with Application Insights
3. Periodically re-evaluate against test set for regression
4. Retrain as new data becomes available

# Iterative Training Workflow

Systematically improve a fine-tuned model through successive experiments.

## The Core Loop

```
1. Train with current config
2. Analyze training curves
3. Evaluate on held-out set
4. Diagnose what to change
5. Plan next experiment
→ Better than baseline? → Good enough? → Ship it (or loop back to 4)
```

**Rule**: Change ONE variable per experiment.

## Experiment Tracking

| Run | Base model | Dataset | Epochs | LR | Batch | Best val_loss | Combined eval |
|-----|-----------|---------|--------|-----|-------|--------------|---------------|
| R1 | gpt-4.1-mini | v1 (335 ex) | 2 | 1.0 | default | 0.320 | 8.05 |
| R2 | gpt-4.1-mini | v1 (335 ex) | 2 | 0.5 | default | 0.310 | 9.15 |
| ... | ... | ... | ... | ... | ... | ... | ... |

## What to Try (Priority Order)

### Priority 1: Data Quality (highest leverage)
- **Fix inconsistencies**: Contradicting examples confuse the model
- **Add diversity**: Add examples for input types the model fails on
- **Reduce noise**: Remove "correct but not ideal" outputs

### Priority 2: Hyperparameters

See `references/hyperparameters.md` for full guide.

**Quick sweep strategy:**
1. Baseline: epochs=2, lr=1.0
2. Overfitting → lr=0.5 or epochs=1
3. Underfitting → lr=1.5 or epochs=3
4. Good LR found → try batch_size=16 or 32

### Priority 3: Base Model

| Model | Best for |
|-------|----------|
| gpt-4.1-mini | Best quality-per-dollar, most tasks |
| gpt-4.1-nano | Fastest inference, simple tasks |
| gpt-oss-20b | Large datasets, lowest absolute loss |
| Ministral-3B | Lightweight, fast inference |
| Qwen-3-32B, Llama-3.3-70B | Multilingual or specialized tasks |

### Priority 4: Training Type
- SFT plateaued + need better reasoning → RFT (if model supports it)
- Need style alignment → DPO
- See `references/training-types.md` before switching

## Diagnostic Decision Tree

```
Training curves healthy (no overfitting)?
├─ Yes
│  ├─ Eval improved? → Refine further
│  └─ Eval same/worse? → Data quality issue — filter or augment
└─ No (overfitting)
   ├─ Earlier checkpoint evals well? → Deploy that checkpoint
   ├─ Not severe → Reduce epochs or lower LR
   └─ Severe (ratio > 2.0)
      ├─ Dataset too small → Add more data
      └─ Dataset large → Lower LR dramatically (0.1-0.3)
```

## When to Stop

1. Beaten baseline by meaningful margin (>5%) and last 3 experiments didn't improve
2. Diminishing returns: each experiment improves < 0.1 points
3. Model is "good enough" for production
4. Budget exhausted (time or money)

## Multi-Model Strategy

Run the same dataset through 2-3 base models:
1. **gpt-4.1-mini** — primary candidate
2. **gpt-oss-20b** — large-dataset specialist (500+ examples)
3. **gpt-4.1-nano** — fast inference option

## Common Mistakes

1. Not establishing a baseline first
2. Changing multiple variables at once
3. Overfitting to the eval set (keep a separate final test set)
4. Ignoring training curves (they tell you what to change next)
5. More data without quality check (lower-quality data often makes things worse)
6. Not cleaning up old deployments (wastes quota and money)

# Quickstart: Fine-Tune Your First Model

6 steps from zero to a fine-tuned model using SFT with synthetic data.

> **Time**: ~20 min active + 1-3 hours training.

## Prerequisites

- Azure AI Foundry project with a deployed model (e.g., `gpt-4.1-mini`)
- Python 3.10+ with `openai` installed
- Project endpoint URL and API key (Foundry portal → Project Settings)

## Step 1: Connect to Your Project

```bash
export OPENAI_BASE_URL="https://<your-resource>.services.ai.azure.com/api/projects/<your-project>/openai/v1/"
export AZURE_OPENAI_API_KEY="<your-key>"
```

```python
from openai import OpenAI
import os

client = OpenAI(base_url=os.environ["OPENAI_BASE_URL"], api_key=os.environ["AZURE_OPENAI_API_KEY"])
resp = client.chat.completions.create(model="gpt-4.1-mini", messages=[{"role": "user", "content": "Hello"}], max_tokens=10)
print(resp.choices[0].message.content)
```

## Step 2: Generate Training Data

```python
import json, re

SYSTEM_PROMPT = "You are a concise technical support agent. Answer in 1-2 sentences."

generation_prompt = """Generate 50 diverse technical support conversations.
Each should have a customer question and an ideal agent response (1-2 sentences).
Cover: password resets, billing, product setup, account changes, shipping, troubleshooting.
Return a JSON array where each element has "question" and "answer" fields."""

resp = client.chat.completions.create(
    model="gpt-4.1-mini", messages=[{"role": "user", "content": generation_prompt}],
    max_tokens=8000, temperature=1.0,
)

content = resp.choices[0].message.content
match = re.search(r'```(?:json)?\s*\n(.*?)\n```', content, re.DOTALL)
json_str = match.group(1) if match else content.strip().strip("`").replace("json\n", "")
examples = json.loads(json_str)

for split, name, rng in [("train", "train.jsonl", examples[:40]), ("val", "val.jsonl", examples[40:])]:
    with open(name, "w") as f:
        for ex in rng:
            f.write(json.dumps({"messages": [
                {"role": "system", "content": SYSTEM_PROMPT},
                {"role": "user", "content": ex["question"]},
                {"role": "assistant", "content": ex["answer"]},
            ]}) + "\n")
```

Validate: `python scripts/validate/validate_sft.py train.jsonl`

## Step 3: Baseline the Base Model

```python
with open("val.jsonl") as f:
    test_examples = [json.loads(line) for line in f][:5]

for ex in test_examples:
    resp = client.chat.completions.create(
        model="gpt-4.1-mini", messages=ex["messages"][:2], max_tokens=200)
    print(f"Q: {ex['messages'][1]['content']}")
    print(f"Expected: {ex['messages'][2]['content']}")
    print(f"Base model: {resp.choices[0].message.content}\n")
```

## Step 4: Upload Data and Submit Job

```python
import time

with open("train.jsonl", "rb") as f:
    train = client.files.create(file=f, purpose="fine-tune")
with open("val.jsonl", "rb") as f:
    val = client.files.create(file=f, purpose="fine-tune")

for _ in range(30):
    if client.files.retrieve(train.id).status == "processed" and client.files.retrieve(val.id).status == "processed":
        break
    time.sleep(10)

job = client.fine_tuning.jobs.create(
    model="gpt-4.1-mini", training_file=train.id, validation_file=val.id,
    suffix="my-first-ft",
    method={"type": "supervised"},
    hyperparameters={"n_epochs": 2, "learning_rate_multiplier": 1.0},
)
print(f"Job submitted: {job.id}")
```

Or via script:
```bash
python scripts/submit_training.py --model gpt-4.1-mini --training-file train.jsonl --validation-file val.jsonl --type sft --suffix my-first-ft --epochs 2
```

## Step 5: Monitor

```bash
python scripts/monitor_training.py --job-id <your-job-id>
```

Or check [Azure AI Foundry portal](https://ai.azure.com) → Fine-tuning → Jobs.

## Step 6: Deploy, Test, and Compare

```bash
python scripts/deploy_model.py --model-id <fine-tuned-model-name> --name my-ft-deployment --capacity 50
```

```python
for ex in test_examples:
    base = client.chat.completions.create(model="gpt-4.1-mini", messages=ex["messages"][:2], max_tokens=200)
    ft = client.chat.completions.create(model="my-ft-deployment", messages=ex["messages"][:2], max_tokens=200)
    print(f"Q: {ex['messages'][1]['content']}")
    print(f"Base:       {base.choices[0].message.content}")
    print(f"Fine-tuned: {ft.choices[0].message.content}\n")
```

## What's Next

- **Scale data**: 200-500 examples → `workflows/dataset-creation.md`
- **Try RFT**: For verifiable answers → `references/training-types.md`
- **Debug**: `workflows/diagnose-poor-results.md`
- **Full guide**: `workflows/full-pipeline.md`

# Agent Optimizer in Foundry — Scaffold Python Agent

Prepare an existing Python hosted agent for Agent Optimizer in Foundry, then run optimization, apply the selected candidate locally, and deploy through azd after review.

## When to Use This Skill

USE FOR: make my Python agent optimizable with Agent Optimizer in Foundry, scaffold optimizer config, add `load_config`, prepare `.agent_configs`, configure eval.yaml, run azd ai agent optimize, apply optimizer candidate, deploy optimized agent.

DO NOT USE FOR: non-Python agents, prompt agents, running standalone batch evaluations, prompt optimization of an already deployed agent, or general Foundry deployment. For normal deployment, use [deploy](../deploy/deploy.md). For eval analysis loops, use [observe](../observe/observe.md).

## Quick Reference

| Property | Value |
| -------- | ----- |
| Phase | Scaffold, optimize, apply locally, deploy |
| Supported language | Python |
| Required runtime | azd project with hosted agent |
| Required package | `azure-ai-agentserver-optimization` |
| Required import | `from azure.ai.agentserver.optimization import load_config` |
| Required baseline | `.agent_configs/baseline/` in the agent's service source directory |
| Supported targets | instruction, model, skill folder, function tool definitions |
| azd setup | [azd Setup](references/azd-setup.md) |
| Detailed scaffold steps | [Scaffold Workflow](references/scaffold.md) |
| Python/file patterns | [Python Patterns](references/python-patterns.md) |
| Eval config | [eval.yaml Guidance](references/eval-yaml.md) |
| Optimize flow | [Optimize Workflow](references/optimize-workflow.md) |

## High-Level Lifecycle

1. **Prepare azd:** Verify azd, login, and `azure.ai.agents` extension with [azd Setup](references/azd-setup.md).
2. **Scaffold:** Follow [Scaffold Workflow](references/scaffold.md) when SDK wiring or `.agent_configs/baseline/` is missing; stop for review if files changed.
3. **Configure eval:** Create or update `eval.yaml` using [eval.yaml Guidance](references/eval-yaml.md).
4. **Optimize:** Run and monitor `azd ai agent optimize` with [Optimize Workflow](references/optimize-workflow.md).
5. **Apply and deploy:** Apply the selected candidate locally, review the diff, then deploy with `azd deploy`.

## Workflow

1. Resolve the target agent root and confirm it is a Python hosted agent.
2. Read [azd Setup](references/azd-setup.md), then [Scaffold Workflow](references/scaffold.md) if scaffolding is needed.
3. Read [eval.yaml Guidance](references/eval-yaml.md) and configure optimization inputs from known dataset/evaluator context.
4. Read [Optimize Workflow](references/optimize-workflow.md), run optimization, and ask before applying a candidate.
5. After local review and approval, deploy with `azd deploy`, then invoke via [invoke](../invoke/invoke.md).

## Guardrails

- Target hosted Python agents only.
- Preserve existing frameworks, tools, hosting adapters, protocols, and entrypoints.
- Do not use one global scaffold across multi-agent roles unless the architecture already has one global prompt/model or the user approves.
- Keep edits scoped to the selected agent root.
- Do not apply candidates or deploy automatically; stop for review first.
- Prefer `azd ai agent optimize apply --candidate` plus `azd deploy` over direct optimize deploy so source changes are reviewable.

# azd Setup

Use this before running Agent Optimizer operations. This skill targets agent code repos that use azd and hosted agents.

## Verify prerequisites

Run from the selected agent repo:

```bash
azd version
az login
azd ai agent --help
azd ai agent optimize --help
```

If `azd ai agent` is unavailable, install or update the `azure.ai.agents` azd extension using the official extension source. If the needed version is private preview only, ask the user for their approved extension source; do not embed private registry commands.

## Resolve hosted-agent context

Use [Common Project Context Resolution](../../../SKILL.md#agent-common-project-context-resolution). Prefer azd context from `azure.yaml` and `azd env get-values`.

Confirm:

- selected service uses `host: azure.ai.agent`
- selected root contains Python agent code
- agent kind is `hosted`
- project endpoint/project ID and deployed agent name/version are known

If the agent's `azure.yaml` service block is missing, ask before initializing:

```bash
azd ai agent init --project-id <project-id>
```

Use the project ID from azd context when available; otherwise ask the user. After init, stop for review of the generated `azure.yaml` service block.

# eval.yaml Guidance

Create `eval.yaml` directly when the conversation or `.foundry/agent-metadata*.yaml` already selected the dataset/evaluators. Otherwise ask whether to run `azd ai agent eval generate` or let optimize use built-in defaults.

## Include

```yaml
name: <suite-or-optimization-name>
agent:
  name: <agent-name>
  kind: hosted
  version: "<agent-version>"
  model: <baseline-model-deployment-name>
  config: .agent_configs/baseline/metadata.yaml
dataset:
  local_uri: <path-to-jsonl>
  # name: <foundry-dataset-name>
  # version: "<dataset-version>"
# validation_dataset:
#   name: <validation-dataset-name>
#   version: "<validation-version>"
evaluators:
  - <evaluator-name>
  - name: <custom-evaluator-name>
    version: "<evaluator-version>"
    local_uri: <local-evaluator-json>
options:
  eval_model: <existing-chat-model-deployment-name>
  optimization_model: <allowed-optimizer-model-deployment-name>
  max_candidates: 4
  optimization_config:
    model_search_space:
      - <target-model-deployment-name>
```

Use existing model deployments for `agent.model` and `options.eval_model`; do not assume `gpt-4o`.

For `options.optimization_model`, first verify that the target Foundry project has a deployment whose name is in this allowlist:

- `GPT-5`
- `GPT-5.1`
- `GPT-5.2`
- `GPT-5.4`
- `GPT-5.5`
- `DeepSeek-V4-Pro`
- `DeepSeek-V-3.2`

If none exist, ask the user to deploy one before configuring optimization. Use `options.optimization_config.model_search_space` only for target model candidates that exist in the project; it may include the baseline model when the user wants it compared.

## Generate evals when inputs are missing

Prefer `eval generate` over older init flows:

```bash
azd ai agent eval generate --dataset <path-to-jsonl>
azd ai agent eval generate --reset-defaults
```

After generation, run `azd ai agent optimize --optimize-model <allowed-optimizer-model-deployment-name>` from the azd project; optimize auto-detects the generated `eval.yaml`.

## Skip

Do not add these fields unless the user explicitly asks and understands the tradeoff:

- `target_attributes`
- `budget`
- `min_improvement`
- `pass_threshold`
- `keep_versions`
- `generation_instruction`
- `max_samples`
- `trace_days`
- legacy `dataset_file`, `dataset_reference`, or `validation_reference` when writing a new file

Keep `target_attributes` omitted so azd can auto-detect optimizable attributes.

## Source mapping

| Source | eval.yaml field |
|--------|-----------------|
| effective azd context | `agent.name`, `agent.version`, `agent.kind` |
| baseline config | `agent.model`, `agent.config` |
| selected local dataset JSONL | `dataset.local_uri` |
| selected remote/local dataset | `dataset.name`, `dataset.version`, `dataset.local_uri` |
| selected validation dataset | `validation_dataset` |
| selected Foundry/local evaluators | `evaluators[]` |
| selected judge/eval deployment | `options.eval_model` |
| selected optimizer deployment | `options.optimization_model` |
| selected target model candidates | `options.optimization_config.model_search_space` |

Treat older `dataset_file`, `dataset_reference`, `validation_reference`, `max_iterations`, and `optimization_config.model` as legacy inputs when reading existing files, but write new files with the current contract above.

# Optimize Workflow

Use this after azd setup and scaffold review are complete.

## 1. Prepare context

1. Resolve the hosted agent with [azd Setup](azd-setup.md).
2. If SDK wiring or `.agent_configs/baseline/` is missing, run [Scaffold Workflow](scaffold.md) first.
3. If scaffolding changed files, stop and ask the user to review before optimization.
4. Ensure `eval.yaml` exists using [eval.yaml Guidance](eval-yaml.md), generate it with `azd ai agent eval generate`, or ask whether to use built-in optimize defaults.
5. Before setting `--optimize-model` or `options.optimization_model`, verify the project has an existing deployment from the allowed optimizer list: `GPT-5`, `GPT-5.1`, `GPT-5.2`, `GPT-5.4`, `GPT-5.5`, `DeepSeek-V4-Pro`, or `DeepSeek-V-3.2`.

When evaluation inputs are not already selected, generate them from a reviewed seed dataset or regenerate defaults:

```bash
azd ai agent eval generate --dataset <path-to-jsonl>
azd ai agent eval generate --reset-defaults
```

## 2. Run optimize

Run from the azd project/agent root:

```bash
azd ai agent optimize --optimize-model <allowed-optimizer-model-deployment-name>
```

If multiple services are detected, let azd prompt or ask the user which service to use. If `eval.yaml` exists or was generated, use it when it matches the selected agent; otherwise ask before regenerating or ignoring it.

## 3. Monitor

Use these when the job is long-running or the user asks:

```bash
azd ai agent optimize status <operation-id> --watch
azd ai agent optimize list
azd ai agent optimize cancel <operation-id>
```

Capture the operation ID, portal URL, scores, and candidate IDs from output.

## 4. Apply locally

Recommend the best candidate, then ask before applying:

```bash
azd ai agent optimize apply --candidate <candidate-id>
```

After apply, show the source diff and summarize changed files, prompts, model/temperature, tools, and skills.

## 5. Deploy after review

In azd environments, prefer local apply plus:

```bash
azd deploy
```

Do not use `azd ai agent optimize deploy --candidate <candidate-id>` unless the user explicitly requests it. Local apply keeps optimized changes visible for source control review.

# Python Agent Optimizer in Foundry Patterns

Use the Azure SDK optimization package and a local baseline folder. The baseline is file-based; call `load_config()` without code-level fallback parameters.

## Install and Import

Add `azure-ai-agentserver-optimization` to `requirements.txt` or the project dependency file:

```text
azure-ai-agentserver-optimization
```

Import from the SDK namespace:

```python
from azure.ai.agentserver.optimization import load_config
```

## Baseline Folder

Create `.agent_configs/baseline/` in the agent's service source directory (beside the entry point):

```text
<agent-root>/
  main.py
  .agent_configs/
    baseline/
      metadata.yaml
      instructions.md
      tools.json
      skills/<skill-name>/SKILL.md
```

Example `metadata.yaml`:

```yaml
model: <existing-chat-model-deployment-name>
temperature: 0.7
instruction_file: instructions.md
skill_dir: skills
tool_file: tools.json
```

`instructions.md` contains the selected baseline system/developer instructions. Include only skill folders relevant to the optimization goal.

Choose a `model` value that already exists as a model deployment in the target Foundry project. Do not assume `gpt-4o` is available.

## Tools File

Use OpenAI function-calling tool objects under top-level `tools`. Currently, only function tool definition optimization is supported:

```json
{
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "lookup_policy",
        "description": "Look up the company travel policy.",
        "parameters": {
          "type": "object",
          "properties": {
            "dept": {
              "type": "string",
              "description": "Department name"
            }
          }
        }
      }
    }
  ]
}
```

## Runtime Wiring

Call `load_config()` with no defaults:

```python
config = load_config()
instructions = config.compose_instructions()
model = config.model
```

For Microsoft Agent Framework:

```python
client = FoundryChatClient(
    project_endpoint=project_endpoint,
    model=config.model,
    credential=credential,
)

agent = Agent(
    client=client,
    instructions=config.compose_instructions(),
    tools=tools,
)
```

Patch optimized function tool definitions through the public helper. It updates matching function docs, descriptions, and parameter descriptions:

```python
config.apply_tool_descriptions(tools)
```

Load skills on demand when the runtime has a safe skill/tool mechanism:

```python
from pathlib import Path
from azure.ai.agentserver.optimization import load_skills_from_dir

skills = load_skills_from_dir(Path(config.skills_dir)) if config.skills_dir else []
```

## Target Selection

Use evaluator and dataset goals to decide what belongs in the baseline:

| Signal | Prefer |
| ------ | ------ |
| `relevance`, `task_adherence` | primary instructions and model |
| `intent_resolution` | router/orchestrator instructions |
| `builtin.tool_call_accuracy` | tool-calling instructions and OpenAI function tool definitions |
| safety/groundedness | safety, retrieval, citation, or answer-synthesis instructions |

For multi-agent apps, scaffold the target role's instructions and related skills/tools. Do not merge unrelated role prompts into one baseline.

## Runtime Config

The SDK reads optimization context from supported runtime sources. Keep `.agent_configs/baseline/` present so default `load_config()` startup has a local baseline. Use `load_config(config_dir="my_configs")` only for non-default local config directories, and `load_config(required=False)` only when the app can intentionally run without optimization config.

## Verification Checklist

- Dependency file includes `azure-ai-agentserver-optimization`
- `from azure.ai.agentserver.optimization import load_config` succeeds
- `.agent_configs/baseline/metadata.yaml` exists and points to existing files
- `load_config()` is called without defaults unless using an intentional `config_dir` or `required=False`
- Changed Python files compile and preserve the hosting adapter/protocol
- User is asked to review before deployment

# Scaffold Workflow

Use this workflow to make a Python agent optimizable before running Agent Optimizer in Foundry.

## Step 1: Resolve Target and Goal

Stay inside the selected agent root. Confirm the project is Python using `requirements.txt`, `pyproject.toml`, `setup.py`, or Python entrypoints.

Identify the optimization goal from user input, selected `evaluationSuites[]`, `.foundry/evaluators/*`, recent result summaries, datasets, or code/test comments. If the goal is unclear, proceed conservatively and explain that evaluator-specific targeting improves optimization quality.

## Step 2: Inventory Safe Targets

Scan for instructions, model selection, skill folders, function tool definitions, topology, and hosting entrypoint. Record file path, symbol/name, role, current value, and whether it is safe to expose through the optimizer.

Classify topology as single-agent, orchestrator/supervisor, specialist tool-agent, peer multi-agent, or unknown runtime. Do not collapse role-specific prompts into one global prompt. Ask before editing when multiple scopes are plausible.

Use [Python Patterns](python-patterns.md#target-selection) to map evaluator/dataset goals to the smallest useful baseline.

## Step 3: Scaffold Baseline Files

Create the required `.agent_configs/baseline/` folder in the agent's service source directory (beside the entry point):

```text
.agent_configs/
  baseline/
    metadata.yaml
    instructions.md
    tools.json
    skills/<skill-name>/SKILL.md
```

`metadata.yaml` points to selected baseline files:

```yaml
model: <existing-chat-model-deployment-name>
temperature: 0.7
instruction_file: instructions.md
skill_dir: skills
tool_file: tools.json
```

Write the selected baseline prompt to `instructions.md`. Include only relevant skills under `skills/`. Use `tools.json` only for OpenAI function-calling tool definitions; see [Python Patterns](python-patterns.md#tools-file).

Choose a `model` value that already exists as a model deployment in the target Foundry project.

Do not use code-level defaults as the optimization baseline.

## Step 4: Install and Wire SDK

Add `azure-ai-agentserver-optimization` to the target agent project's dependency file:

```text
azure-ai-agentserver-optimization
```

Wire the agent with no default parameters:

```python
from azure.ai.agentserver.optimization import load_config

config = load_config()
```

Map resolved values:

- Instructions -> `config.compose_instructions()`
- Model -> `config.model`
- Skills -> `config.skills_dir` with `load_skills_from_dir(...)` only when the runtime has a safe skill/tool mechanism
- Function tool definitions -> `config.apply_tool_descriptions(tools)` when tool metadata can be patched safely

Do not add optimization runtime env vars to the agent's `environmentVariables` in `azure.yaml`. The default local config path is `.agent_configs/`; use `load_config(config_dir="...")` only when the scaffold intentionally uses a non-default local config directory.

## Step 5: Verify and Stop

Run Python syntax checks, SDK import smoke test, baseline config smoke test with no-arg `load_config()`, workspace diagnostics, and cheap relevant project tests.

End with a review checkpoint. Summarize changed files, optimization targets, evaluator goals, global side effects, and verification. Do not deploy automatically.

After user review, continue with [Optimize Workflow](optimize-workflow.md).

# Create Hosted Agent (azd ai)

Scaffold a hosted Foundry agent project with the Azure Developer CLI (`azd`) and the `azure.ai.agents` extension. The same flow covers greenfield (from a curated sample) and brownfield (lift existing code), then drops you into a local inner-loop so you can iterate before deploying.

> **Creating a new agent end-to-end from scratch?** Use [quick-start-hosted.md](quick-start-hosted.md) instead -- an opinionated happy-path with safe defaults. Stay here for anything not covered by the quickstart.

> **Scope:** `azd ai` is the preferred *code-first* path -- use it when the intent is agent code on disk, in a repo, with infrastructure-as-code and a local inner-loop. If the intent is only to create a remote agent resource (no code on disk), other approaches may apply -- for prompt agents see [create-prompt.md](create-prompt.md), or use the Foundry MCP tools / portal.

## Quick Reference

| Property | Value |
|----------|-------|
| Agent type | Hosted (container or code) |
| Primary CLI | `azd ai agent` (from extension `azure.ai.agents`) |
| Scaffold command | `azd ai agent init -m <manifestUrl> --deploy-mode code --runtime python_3_13 --entry-point main.py`, pass `--runtime dotnet_10 --entry-point MyAgent.dll` for .NET project (or `--src <dir>` for brownfield) |
| Local run | `azd ai agent run` + `azd ai agent invoke --local "..."` |
| Deploy handoff | [deploy/deploy.md](../deploy/deploy.md) |
| Sample catalog | `azd ai agent sample list --featured-only --output json` |
| Reference docs | [azd-ai-cli](references/azd-ai-cli.md), [local-run](references/local-run.md), [tools](references/tools.md) |

## When to Use This Skill

- Create a new hosted agent from a curated Foundry sample.
- Lift an existing agent project (Python, .NET) into a hosted Foundry agent.
- Add tools (web search, AI Search, MCP, A2A) to a hosted agent.
- Run and iterate on a hosted agent locally before deploying.

For prompt agents (LLM + instructions, no container), use [create-prompt.md](create-prompt.md). For deploy, use [deploy.md](../deploy/deploy.md).

## Hosted vs Prompt

| | Hosted | Prompt |
|--|--------|--------|
| Custom Python / .NET code? | Yes -> this skill | No -> [create-prompt.md](create-prompt.md) |
| Tools / RAG / MCP / A2A | Toolbox + connections | Built-in tool configs |
| Local debugging | `azd ai agent run` | Limited |
| Output | New immutable agent version per `azd deploy` | `agent_update` via MCP / SDK |

## Workflow

### Step 1 -- Verify the environment

Two pre-flight checks — run each script and act on its `[OK]` / `[WARN]` / `[ACTION]` summary prefixes.

**1a — Canvas-first entry (GitHub Copilot app).** Detects whether the runtime is the GitHub Copilot app (`AI_AGENT=github_copilot_app_agent`) and the Foundry Agent Canvas extension is installed. If both are true, the canvas must be opened first so the user can authenticate and select a Foundry project before scaffolding. Run this check first (it can short-circuit the rest). **Skip 1a** if the user opts out, e.g. "skip the canvas" / "use the CLI".

```bash
./scripts/check-canvas-entry.sh     # macOS / Linux
./scripts/check-canvas-entry.ps1    # Windows (pwsh)
```

- **No `[ACTION]`** (only `[OK]`/`[WARN]`) — the gate doesn't apply (not in the Copilot app, or the canvas isn't installed). Continue to 1b.
- **`[ACTION]`** — the gate applies. If the user message's `<canvas-context>` already lists **`canvas="agent-builder"`**, the user has already driven the canvas — continue to 1b using their prompt/config (loop guard, independent of prompt wording). Otherwise `open_canvas` (`canvasId: "agent-builder"`), ask the user to **create the agent from the canvas: sign in, select a subscription + Foundry project, then Send**, then **stop — do not run 1b or scaffold**.

**1b — Tooling & auth.** Run the bundled verification script before any other create/deploy command:

```bash
./scripts/verify-environment.sh     # macOS / Linux
./scripts/verify-environment.ps1    # Windows (pwsh)
```

Do not continue past Step 1 while any `[ACTION]` remains. Never run `az login` or `azd auth login` for the user. Missing authentication is a hard stop before any `azd init`, `azd ai agent init`, `azd provision`, `azd deploy`, or other deploy command.

Act on the summary prefixes:

- `[OK]` -- nothing to do.
- `[WARN]` -- non-blocking; continue.
- `[ACTION]` -- resolve first, then rerun the script. If `az` or `azd` is missing, ask before installing in interactive mode; install directly in non-interactive mode. For how to install `azd`, see <https://learn.microsoft.com/en-us/azure/developer/azure-developer-cli/install-azd>. In any mode, never run `az login` or `azd auth login`; stop and ask the user to log in manually. Missing `azure.ai.agents` / `azure.ai.projects` extensions may be resolved with `azd extension install <name>`. Failed `az` or `azd` auth checks must stop the workflow until the user logs in manually.

> **Preflight: get `AZURE_SUBSCRIPTION_ID` + `AZURE_LOCATION` into the azd env *before* the first `azd ai agent init`.** Without both, init defers model resolution -> `azure.yaml services.ai-project.deployments[]` ends up empty -> `AI_PROJECT_DEPLOYMENTS=[]` -> `azd provision` creates zero model deployments -> the agent service's `environmentVariables` keep the literal `{{AZURE_AI_MODEL_DEPLOYMENT_NAME}}` placeholder. `azd ai agent init` itself has **no** `--subscription` / `--location` flags (those live on core `azd init`). Pick the **first** option that fits, ranked best-first:
>
> 1. **Pre-bootstrap with core `azd init`** — per-project, no global state. **Recommended default for scripted / MCP / agent-driven flows.** Run in the target empty directory:
>    ```bash
>    azd init -t Azure-Samples/azd-ai-starter-basic . -e <env-name> --subscription <id> -l <region>
>    azd ai agent init -m <manifest-url> --no-prompt --deploy-mode code --runtime python_3_13 --entry-point main.py
>    ```
>    Core `azd init` creates `azure.yaml` + the azd env with `AZURE_SUBSCRIPTION_ID` / `AZURE_LOCATION` already populated; the extension's `ensureProject` sees the existing project and the model resolver reads the values core just wrote. (Use this even though `azd ai agent init` can scaffold from scratch — it's the only headless path that avoids deferral without mutating global config.)
> 2. **`azd ai agent init --project-id <arm-id>`** — only when the Foundry project already exists in Azure. Init extracts the subscription from the ARM ID and uses the project's own location. Skip Option 1.
> 3. **Interactive mode** — omit `--no-prompt`. Init prompts for subscription + location. Only when a human is at a terminal.
> 4. **Global config (last resort, mutates `~/.azure/config.json` for every azd project on the machine):**
>    ```bash
>    azd config set defaults.subscription <id>
>    azd config set defaults.location <region>
>    ```
>    Avoid in per-project / scripted flows. Use only when no per-project option fits and the machine is single-tenant.
>
> **If you only discover the need to set sub + location *after* init has already scaffolded `src/<name>/`, do *not* naively re-run `azd ai agent init`.** It is not idempotent: under `--no-prompt` it silently creates `<service>-2`; in interactive mode the collision prompt's **default selection is "Use a different service name"** (you must actively arrow-up to "Overwrite existing"). See the [recovery paths](#step-4a----greenfield-scaffold-from-a-sample) in Step 4a.
>
> Never `azd env set AI_PROJECT_DEPLOYMENTS '[...]'` and never `az cognitiveservices account deployment create ...` for the azd Golden Path — both break the lifecycle.

Branch on the reported agent status:

- `not_deployed` -> Step 2.
- `active` / `deployed` -> already deployed. Skip to [deploy/deploy.md](../deploy/deploy.md) for redeploy or [tools](references/tools.md) to add a tool.

### Step 2 -- New or existing Foundry project?

Ask: "Do you want to create a new Foundry project, or use an existing one?" Skip the question when the prompt already says to use an existing project or supplies a Foundry project endpoint / project ARM resource ID.

- **New project** -- do NOT pass `--project-id`. `azd provision` (in deploy) will create it.
- **Existing project with ARM resource ID** -- pass that exact ID to `azd ai agent init --project-id`.
- **Existing project with Foundry project endpoint only** -- resolve the project ARM resource ID with the bundled script, then pass the returned `id` to `azd ai agent init --project-id`:
  ```bash
  ./scripts/resolve-project-id.sh --endpoint "<foundry-project-endpoint>"     # macOS / Linux
  ./scripts/resolve-project-id.ps1 -Endpoint "<foundry-project-endpoint>"     # Windows (pwsh)
  ```
- **Existing project with neither endpoint nor ARM ID** -- ask for the ARM resource ID.

Do not guess, derive, or construct the project ID from the endpoint. For `--project-id`, pass either the user-supplied project ARM resource ID or the `id` returned by Azure lookup / the bundled resolve script.

### Step 3 -- Pick the scaffolding source

| User has ... | Use |
|--------------|-----|
| Empty workspace, or wants a starter | **Greenfield** -- Step 4a |
| Hand-written agent code already in cwd | **Brownfield** -- Step 4b |

If unsure, default to greenfield. Never guess a manifest URL by hand.

### Step 4a -- Greenfield: scaffold from a sample

List the curated catalog (filter by language if known):

```bash
azd ai agent sample list --featured-only --language python --output json
```

Each entry has a `manifestUrl` and an `initCommand`. Prefer direct code deploy at init time. `--no-prompt` defaults to container deploy unless you pass `--deploy-mode code`, so include the code flags up front.

For a generic new hosted agent request, start from the basic sample. Use tool/function-calling samples only when the user explicitly asks for external actions, APIs, tools, connectors, or data lookup.

> **Before running init**, make sure subscription + location are resolvable via one of the four options in [Step 1 preflight](#step-1----verify-the-environment). For headless / scripted flows the recommended path is to **pre-bootstrap with core `azd init`**:
>
> ```bash
> azd init -t Azure-Samples/azd-ai-starter-basic . -e <env-name> --subscription <id> -l <region>
> ```
>
> Then run `azd ai agent init` inside the bootstrapped directory. `azd ai agent init` itself has **no** `--subscription` / `--location` flags (passing them fails with `unknown flag`); core `azd init` does. If init still defers resolution (empty `services.ai-project.deployments[]` / `{{...}}` placeholder), see the recovery paths after the init example below — do **not** blindly re-run init.

Python Example (add `--project-id "<resourceId>"` for an existing Foundry project; add `--agent-name <name>` if the user wants a custom name -- omit otherwise to keep the sample default):

```bash
azd ai agent init --no-prompt \
  -m "<manifestUrl>" \
  --deploy-mode code \
  --runtime python_3_13 \
  --entry-point main.py
```

> `--agent-name` at init sets both the `azure.yaml` service key and its `name:` in one shot; renaming after init requires editing both in `azure.yaml`.

Do not run `azd env new`, `azd env select`, or `azd env set` before `azd ai agent init` in a new temp/workspace; there is no azd project yet, so those commands fail and waste time. For an existing project, `--project-id` is enough during init. Set endpoint/model values immediately after init, once `azure.yaml` and the azd env exist.

> Tip: if the manifest declares a `parameters:` block (check by `curl <manifestUrl>`), collect required values before init when an azd project already exists. In a new empty workspace, prefer a sample without required secrets; there is no azd env to set until init creates the project files.

`init` writes `azure.yaml` (or appends the agent service to it), the agent source under `src/<name>/`, and `<service-dir>/.agentignore` (code-deploy only). A successful direct-code init produces an `azure.yaml` service block (`host: azure.ai.agent`) with `codeConfiguration:`. For file shapes, see [azd-ai-cli](references/azd-ai-cli.md).

#### Model deployments (azd Golden Path)

`azure.yaml services.ai-project.deployments[]` is the **single source of truth** for model deployments in azd-managed Foundry projects. Model deployments live under the dedicated `ai-project` service (`host: azure.ai.project`); the agent service links to it via `uses: [ai-project]` and references the model through its `environmentVariables`. The flow is:

```
manifest → azd ai agent init → azure.yaml ai-project deployments[] → AI_PROJECT_DEPLOYMENTS env (internal) → Bicep → Microsoft.CognitiveServices/accounts/deployments
```

Rules:

- **`azd ai agent init` writes `services.ai-project.deployments[]` from the sample's manifest** and also sets `AZURE_AI_MODEL_DEPLOYMENT_NAME` to the first deployment's `name`. `azd provision` then creates the deployment through Bicep. No `az` calls are needed in the Golden Path.
- **`deployments[].name` is the literal Azure deployment resource name** — not a label, not a placeholder. Use a human-readable model name (e.g. `gpt-4o-mini`, `gpt-4.1-mini`). **Never** use the literal string `AZURE_AI_MODEL_DEPLOYMENT_NAME` as the `name` value; doing so creates a deployment literally named `AZURE_AI_MODEL_DEPLOYMENT_NAME` and the agent will 404 on its first invoke.
- **Adding a *second* model (or any change to `services.ai-project.deployments[]`) to an existing project:** edit `azure.yaml services.ai-project.deployments[]` directly (and update the agent service's `environmentVariables` `AZURE_AI_MODEL_DEPLOYMENT_NAME` if the new entry should become the default), then run `azd provision`. The extension's `preprovision` hook calls `envUpdate` automatically, which re-marshals the deployments and re-writes `AI_PROJECT_DEPLOYMENTS` with the correct double-escaping before Bicep runs. **Do not re-run `azd ai agent init`** for this case — it triggers the non-idempotent collision flow (see anti-patterns) and at best (with explicit "Overwrite existing") re-resolves models from the original manifest rather than merging your edit.
- **Agent `environmentVariables`: prefer `${AZURE_AI_MODEL_DEPLOYMENT_NAME}` over a hardcoded model name.** The `${VAR}` form is resolved from the active azd env at run / deploy time, so a single `azd env set AZURE_AI_MODEL_DEPLOYMENT_NAME <name>` (or env switch dev → prod) updates the agent without touching the file. Init writes this form by default; only the literal `{{AZURE_AI_MODEL_DEPLOYMENT_NAME}}` (double braces) is a failure marker that means model resolution deferred.
- **Recovery: `services.ai-project.deployments[]` is empty or the agent service's `environmentVariables` have the literal `{{AZURE_AI_MODEL_DEPLOYMENT_NAME}}` placeholder.** First get sub + location into the env (see [Step 1 preflight](#step-1----verify-the-environment) options). Then pick **one** of these three paths — init is **not** idempotent:
  1. **Clean re-init (preferred when no user code has been added to `src/<name>/` yet):** delete `src/<name>/`, remove the `services.<name>:` block from `azure.yaml`, then re-run `azd ai agent init`. No collision, scaffolds cleanly with the resolved model.
  2. **Interactive overwrite:** re-run `azd ai agent init` **without `--no-prompt`**. When the collision prompt appears, **actively arrow-up and select "Overwrite existing"** — the default selection is *not* overwrite (it's "Use a different service name", which produces `<name>-2`).
  3. **Hand-fix in place (preserves any user code in `src/<name>/`):** edit `azure.yaml services.ai-project.deployments[]` to add the model block (`name`, `model.{name, format, version}`, `sku.{name, capacity}`), replace the literal `{{AZURE_AI_MODEL_DEPLOYMENT_NAME}}` in the agent service's `environmentVariables` with `${AZURE_AI_MODEL_DEPLOYMENT_NAME}`, then `azd env set AZURE_AI_MODEL_DEPLOYMENT_NAME <deployment-name>`. Run `azd provision`; the `preprovision` hook auto-syncs `AI_PROJECT_DEPLOYMENTS`.
- **Anti-patterns — do not do these:**
  - **Blindly re-running `azd ai agent init` against an existing project.** Under `--no-prompt` init silently auto-suffixes (`<service>-2`, then `-3`, ...) via `nextAvailableName`; in interactive mode the collision prompt's default is "Use a different service name". There is **no flag** (`--force` does not apply here) to make `--no-prompt` overwrite. Use one of the three recovery paths above.
  - **Reaching for `azd config set defaults.subscription` / `defaults.location` as the *first* fix for the deferral.** This mutates `~/.azure/config.json` for every azd project on the machine. Prefer pre-bootstrap with `azd init -t ... --subscription -l` (per-project) or `--project-id` (existing project) first — see the [Step 1 preflight options](#step-1----verify-the-environment).
  - `azd env set AI_PROJECT_DEPLOYMENTS '[...]'` — `AI_PROJECT_DEPLOYMENTS` is internal extension state. The extension writes it with double-escaped JSON (`\\` and `\"`) required by Bicep parameter substitution; `azd env set` only single-escapes and breaks the parse with `invalid character 'n' after object key:value pair`.
  - `az cognitiveservices account deployment create ...` against the azd-managed Foundry account — creates the deployment outside the azd lifecycle, so `azd provision` won't manage it and `azd down` won't clean it up. Use `az cognitiveservices` (or [models/deploy-model](../../models/deploy-model/SKILL.md)) **only** for shared/pre-existing Foundry projects that are not managed by this azd project.
  - Hand-patching the `{{AZURE_AI_MODEL_DEPLOYMENT_NAME}}` placeholder in the agent service's `environmentVariables` *without also* adding the matching entry to `azure.yaml services.ai-project.deployments[]` — the agent will reference a deployment name that Bicep never created. Use the [hand-fix recovery path](#step-4a----greenfield-scaffold-from-a-sample) above (path #3) which fixes both together.

Check the scaffold before local run:

1. **Verify `azure.yaml services.ai-project.deployments[]` is non-empty** and that the agent service's `environmentVariables` `AZURE_AI_MODEL_DEPLOYMENT_NAME` is a literal value or the `${AZURE_AI_MODEL_DEPLOYMENT_NAME}` substitution form — **not** the double-brace literal `{{AZURE_AI_MODEL_DEPLOYMENT_NAME}}` (that placeholder is the marker that init deferred model resolution). Also confirm `azure.yaml` has only **one** service entry for your agent — a duplicate `<name>-2` means a previous init re-ran against the existing project (collision prompt default + `--no-prompt` silent auto-suffix; see anti-patterns above). If either condition fails, use one of the three [recovery paths in the anti-patterns section](#model-deployments-azd-golden-path) (clean re-init / interactive overwrite / hand-fix). Do **not** `azd env set AI_PROJECT_DEPLOYMENTS`.
2. If the user supplied an existing project endpoint, project ARM ID, or model deployment name, set them in the active azd env and verify the values. `azd ai agent run` injects azd env values before `.env`, so a stale `AZURE_AI_MODEL_DEPLOYMENT_NAME` can override a correct `.env` file.
   ```bash
   azd env set AZURE_AI_PROJECT_ENDPOINT "<project-endpoint>"
   azd env set AZURE_AI_PROJECT_ID "<project-arm-id>"
   azd env set AZURE_AI_MODEL_DEPLOYMENT_NAME "<model-deployment-name>"
   azd env get-values
   ```
3. Create the agent source `.env` with the same endpoint and model deployment values:
   ```env
   FOUNDRY_PROJECT_ENDPOINT=https://<account>.services.ai.azure.com/api/projects/<project>
   AZURE_AI_MODEL_DEPLOYMENT_NAME=<model-deployment-name>
   ```
4. Prefer direct code deployment. Inspect the agent's `azure.yaml` service block; if `codeConfiguration:` is missing and the agent does not need a custom Dockerfile or system packages, add it before deployment.
5. Prefer `--agent-name` at init time (above). Fallback only: if init already ran without it, rename the `azure.yaml` service key AND its `name:` to the same value, preserving its `project:` path.
6. If you change CPU or memory, set it in the agent service's `container.resources` in `azure.yaml`.

### Step 4b -- Brownfield: lift existing code

Use ONLY when the workspace already contains hand-written agent source.

```bash
azd ai agent init --no-prompt \
  --src ./src/my-agent \
  --agent-name my-agent \
  --deploy-mode code \
  --runtime python_3_13 \
  --entry-point app.py
```

`--runtime` and `--entry-point` are required with `--deploy-mode code --no-prompt`. Runtimes: `python_3_13`, `python_3_14`, `dotnet_10`. `--deploy-mode container` builds from `Dockerfile`. For an existing Foundry project, add `--project-id "<resourceId>"`.

### Step 5 -- Write the agent instruction file (required)

Applies to both 4a and 4b, and to any existing project you touch (not just fresh scaffolds). Do **not** skip this. Add a one-line marker to the project's `AGENTS.md` (or `CLAUDE.md` in Claude Code) so future prompts reload this skill. If the file already exists, append the line; don't overwrite it or add a second file.

```
This project was built with the microsoft-foundry skill. Before working on or answering questions about foundry agents, read the microsoft-foundry skill first.
```

### Step 6 -- Run locally and iterate

Read and follow [local-run](references/local-run.md). Complete one representative local invocation before deploying.

### Step 7 -- Add capabilities (optional)

Optionally add toolboxes (tools), guardrails (content safety), and skills (behavioral guidelines) before deploying.

### Step 7a -- Add tools (optional)

Tools attach through **toolboxes** -- bundled MCP-compatible endpoints.

#### Toolbox creation boundary

> 🚦 **Creating the toolbox/connection:** write the **full agent-side code** that consumes the toolbox (not just an env var). The **only** thing you leave out is the toolbox configs (for example name/endpoint/connection): put placeholders there, and explicitly tell the user to create the toolbox/connection in [Foundry Toolkit (VS Code)](https://code.visualstudio.com/docs/intelligentapps/tool-catalog) or the [Foundry Portal](https://ai.azure.com/) and write the real values back. Create the toolbox/connection yourself only when the user explicitly asks you to (or supplies the real values).

Flow (only when the user asks you to create the toolbox):

1. Create the **connection** (`azd ai connection create ...`).
2. Create or update the **toolbox** (`azd ai toolbox create` / `connection add`).
3. Set the agent env var (`azd env set TOOLBOX_<NAME>_MCP_ENDPOINT ...`).
4. Reference it in the agent service's `environmentVariables` in `azure.yaml`.
5. `azd deploy`.

Full recipes (GitHub MCP, Azure AI Search, A2A, Bing Custom) in [tools](references/tools.md).

### Step 7b -- Add guardrails (optional)

Attach a content-safety guardrail to the agent or its toolbox. See [guardrail-manage](references/guardrails/guardrail-manage.md) for creating policies and [guardrail-attach](references/guardrails/guardrail-attach.md) for wiring them to agents, model deployments, or toolboxes.

### Step 7c -- Add skills (optional)

Attach reusable behavioral guidelines (skills) to the agent via the toolbox. See [skill-manage](references/skills/skill-manage.md) for creating and versioning skills, [skill-toolbox-attach](references/skills/skill-toolbox-attach.md) for attaching skills to a toolbox, and [skill-attach](references/skills/skill-attach.md) for consuming skills in agent code.

### Step 8 -- Hand off to deploy

Once local invocation succeeds, tell the user the agent is ready and ask if they want to deploy. Read [deploy/deploy.md](../deploy/deploy.md).

## Expected env-var fingerprint (post-provision)

After `azd provision` completes for an `azd ai agent`-scaffolded project (default Basic Agent Setup), `azd env get-values` should show this canonical state. Verify before debugging deployment or runtime issues.

| Variable | Expected value | Notes |
|----------|----------------|-------|
| `ENABLE_HOSTED_AGENTS` | `true` | Set automatically by `azd ai agent init`. |
| `ENABLE_CAPABILITY_HOST` | `false` | Set automatically by `azd ai agent init`. Leave as-is unless you are intentionally targeting Standard Agent Setup. |
| `FOUNDRY_PROJECT_ENDPOINT` | `https://<account>.services.ai.azure.com/api/projects/<project>` | Populated by provision (or pre-set if reusing an existing project). |
| `AZURE_AI_PROJECT_ID` | Full ARM resource ID of the Foundry project | Populated by provision; required for deploy. |
| `AZURE_AI_MODEL_DEPLOYMENT_NAME` | Model deployment name (e.g. `gpt-4o`) | Set automatically by `azd ai agent init` from the first entry in `azure.yaml services.ai-project.deployments[]`. Required for local run and deploy. |
| `AI_PROJECT_DEPLOYMENTS` | escaped JSON array, e.g. `[{\"name\":\"gpt-4o\",...}]` | **Internal extension state.** Managed by `azd ai agent init` from `azure.yaml services.ai-project.deployments[]`. Carries deployments into the Bicep parameter `aiProjectDeploymentsJson`. **Never** set with `azd env set` — manual edits single-escape the JSON and break Bicep `json()` parsing. |
| `AI_AGENT_PENDING_PROVISION` | *(empty / unset)* | Non-empty means provision is still mid-flight; do not deploy. |

`Microsoft.CognitiveServices/accounts/capabilityHosts/agents` is **not** provisioned by `azd ai agent init` (Basic Agent Setup). Its absence is expected. The resource only appears under Standard Agent Setup, which is documented separately in [references/standard-agent-setup.md](../../references/standard-agent-setup.md).

Both `ENABLE_HOSTED_AGENTS` and `ENABLE_CAPABILITY_HOST` are set automatically by `azd ai agent init` — you do not need to manage them. If you ever set them manually outside this flow, see [project/create/create-foundry-project.md](../../project/create/create-foundry-project.md#step-3-create-directory-and-initialize) for the manual-flag procedure.

See the canonical env-var registry: [azure-dev/cli/azd/docs/environment-variables.md](https://github.com/Azure/azure-dev/blob/main/cli/azd/docs/environment-variables.md).

## Common Guidelines

1. **Sample-first** -- always get `manifestUrl` from `azd ai agent sample list`.
2. **Prefer azd over az** -- fall back to `az` only as a last resort, with explicit consent.
3. **Don't auto-login** -- `az login` and `azd auth login` are user-owned browser flows; ask the user and stop.
4. **JSON output** -- add `--output json` only to read-only `azd ai agent` commands such as `show`. Do not add it to `azd ai agent invoke`; invoke supports `default` and `raw`, not `json`.
5. **One file** -- the agent is defined as a service block in `azure.yaml` (`host: azure.ai.agent`). See [azd-ai-cli](references/azd-ai-cli.md).
6. **Reserved env vars** -- `FOUNDRY_*` and `AGENT_*` are platform-injected at runtime; `AI_PROJECT_DEPLOYMENTS`, `AI_PROJECT_RESOURCES`, and `AI_PROJECT_TOOL_CONNECTIONS` are extension-managed transport for Bicep. Never set any of these with `azd env set` -- edit `azure.yaml` and re-run `azd ai agent init`.

## Non-Interactive / YOLO Mode

> Even in `--no-prompt` / `--yolo` mode, don't skip these two:
> - **Project:** if the user named a project or asked to create one, go ahead; otherwise stop and ask before provisioning.
> - **Toolbox/connection:** create it only when the user asked you to; otherwise leave the configs as placeholders and ask.

Defaults when unspecified: greenfield + Python + `azd ai agent sample list --featured-only --language python`, choose the simplest recommended sample that matches the request, plus `--no-prompt` on every write. If creating a new project and the user did not provide a project name, auto-generate one using the pattern `ai-project-<random>` (6-8 lowercase alphanumeric characters). Show the generated name to the user but do not block on confirmation. If using an existing project, ensure `azd ai agent init` receives `--project-id`: use the supplied ARM ID, or run the Step 2 resolve script for the supplied Foundry project endpoint and pass the returned `id`. If the user did not ask to create a new project and did not supply an existing one (ARM ID / endpoint), stop and ask which to use before provisioning. If `az` or `azd` is missing, ask before installing in interactive mode; install directly in non-interactive mode. In any mode, never run `az login` or `azd auth login`; stop and ask the user to log in manually before re-running Step 1. If the manifest declares secret parameters, collect them with `ask_user` and set them via `azd env set PARAM_...` before init -- keep `--no-prompt` (do not fall into azd's interactive prompts).

## Error Handling

| Error | Fix |
|-------|-----|
| `extension not installed` | `azd extension install azure.ai.agents` |
| `not_logged_in` / `login_expired` | Ask user to run `az login` and `azd auth login`; never run those commands for them. |
| `unknown flag: --subscription` / `--location` on `azd ai agent init` | Wrong command — those flags live on **core** `azd init`. See [Step 1 preflight](#step-1----verify-the-environment) for the four options. |
| `no project exists; to create a new project, run azd init` on `azd env set` | The azd env does not exist yet — `azd env set` cannot create it. See [Step 1 preflight](#step-1----verify-the-environment). |
| the agent service's `environmentVariables` contain literal `{{AZURE_AI_MODEL_DEPLOYMENT_NAME}}` placeholder after init | Init deferred model resolution. **Do not blindly re-run init** (default prompt = `<name>-2`; `--no-prompt` silently auto-suffixes). Pick one of the three [recovery paths](#model-deployments-azd-golden-path): clean re-init after deleting `src/<name>/`, interactive overwrite, or hand-fix `azure.yaml` + replace `{{...}}` with `${AZURE_AI_MODEL_DEPLOYMENT_NAME}` and `azd env set AZURE_AI_MODEL_DEPLOYMENT_NAME <name>`, then `azd provision`. |
| `azure.yaml` has duplicate `<service>-2` entry after re-running init | Init is not idempotent: interactive default is "Use a different service name" and `--no-prompt` silently appends `-2`. To recover, delete the `<service>-2` entry from `azure.yaml`, remove `src/<service>-2/`, then `azd provision`. |
| `invalid character 'n' after object key:value pair` during `azd provision` | You used `azd env set AI_PROJECT_DEPLOYMENTS '[...]'` (single-escaped JSON breaks Bicep `json()`). Clear it (`azd env set AI_PROJECT_DEPLOYMENTS ""`), declare the deployment in `azure.yaml services.ai-project.deployments[]` instead, then re-run `azd provision` (its `preprovision` hook re-syncs `AI_PROJECT_DEPLOYMENTS` with the correct double-escaping). |
| `missing_project_endpoint` | Run `azd provision`, or `azd env set AZURE_AI_PROJECT_ENDPOINT <url>` |
| `project_not_found` | cwd has no `azure.yaml`; move to project root or run init |
| Secret parameter prompt under `--no-prompt` | In an empty workspace, choose a simpler sample without secret parameters. In an existing azd project, set `PARAM_<CONN>_<KEY>` with `azd env set` before init; keep `--no-prompt`. |
| `cannot use --version with --local` | Drop `--version`, or drop `--local` to hit the deployed agent |
| `could not detect project type` | Set `startupCommand` in `azure.yaml` or pass `--start-command` |
| Local run issue | Follow [local-run](references/local-run.md) common failures |

Run `azd ai agent doctor --output json` to surface failing checks with `suggestion` fields.

## Next Steps

- Deploy to Foundry -> [deploy/deploy.md](../deploy/deploy.md)
- Add tools -> [tools](references/tools.md)
- Invoke the deployed agent -> [invoke/invoke.md](../invoke/invoke.md)
- Evaluate / optimize -> [observe/observe.md](../observe/observe.md)
- Diagnose failures -> [troubleshoot/troubleshoot.md](../troubleshoot/troubleshoot.md)

# Create Prompt Agent

Create and manage prompt agents in Azure Foundry Agent Service using MCP tools or Python SDK. For hosted agents (container-based), see [create-hosted.md](create-hosted.md).

## Quick Reference

| Property | Value |
|----------|-------|
| **Agent Type** | Prompt (`kind: "prompt"`) |
| **Primary Tool** | Foundry MCP server (`foundry_agents_*`) |
| **Fallback SDK** | `azure-ai-projects` v2.x preview |
| **Auth** | `DefaultAzureCredential` / `az login` |

## Workflow

```
User Request (create/list/get/update/delete agent)
    │
    ▼
Step 1: Resolve project context (endpoint + credentials)
    │
    ▼
Step 2: Try MCP tool for the operation
    │  ├─ ✅ MCP available → Execute via MCP tool → Done
    │  └─ ❌ MCP unavailable → Continue to Step 3
    │
    ▼
Step 3: Fall back to SDK
    │  Read references/sdk-operations.md for code
    │
    ▼
Step 4: Execute and confirm result
```

### Step 1: Resolve Project Context

The user needs a Foundry project endpoint. Check for:

1. `PROJECT_ENDPOINT` environment variable
2. Ask the user for their project endpoint
3. Use `foundry_resource_get` MCP tool to discover it

Endpoint format: `https://<resource>.services.ai.azure.com/api/projects/<project>`

### Step 2: Create Agent (MCP — Preferred)

For a **prompt agent**:
- Provide: agent name, model deployment name, instructions
- Optional: tools (code interpreter, file search, function calling, web search, Bing grounding, memory)

For a **workflow**:
- Workflows are created in the Foundry portal visual builder
- Use MCP to create the individual agents that participate in the workflow
- Direct the user to the Foundry portal for workflow assembly

### Step 3: SDK Fallback

If MCP tools are unavailable, use the `azure-ai-projects` SDK:
- See [SDK Operations](references/sdk-operations.md) for create, list, update, delete code samples
- See [Agent Tools](references/agent-tools.md) for adding tools to agents

### Step 4: Add Tools (Optional)

> ⚠️ **MANDATORY:** Before configuring any tool, **read its reference documentation** linked below to understand prerequisites, required parameters, and setup steps. Do not attempt to add a tool without first reviewing its reference.

| Tool Category | Reference |
|---------------|-----------|
| Code Interpreter, Function Calling | [Simple Tools](references/agent-tools.md) |
| File Search (requires vector store) | [File Search](references/tool-file-search.md) |
| Web Search (default, no setup needed) | [Web Search](references/tool-web-search.md) |
| Bing Grounding (explicit request only) | [Bing Grounding](references/tool-bing-grounding.md) |
| Azure AI Search (private data) | [Azure AI Search](references/tool-azure-ai-search.md) |
| MCP Servers | [MCP Tool](references/tool-mcp.md) |
| Memory (persistent across sessions) | [Memory](references/tool-memory.md) |
| Connections (for tools that need them) | [Project Connections](../../project/connections.md) |

> ⚠️ **Web Search Default:** Use `WebSearchPreviewTool` for web search. Only use `BingGroundingAgentTool` when the user explicitly requests Bing Grounding.

## Error Handling

| Error | Cause | Resolution |
|-------|-------|------------|
| Agent creation fails | Missing model deployment | Deploy a model first via `foundry_models_deploy` or portal |
| MCP tool not found | MCP server not running | Fall back to SDK — see [SDK Operations](references/sdk-operations.md) |
| Permission denied | Insufficient RBAC | Need `Foundry User` role on the project |
| Agent name conflict | Name already exists | Use a unique name or update the existing agent |
| Tool not available | Tool not configured for project | Verify tool prerequisites (e.g., Bing resource for grounding) |
| SDK version mismatch | Using 1.x instead of 2.x | Install `azure-ai-projects --pre` for v2.x preview |
| Tenant mismatch | MCP token tenant differs from resource tenant | Fall back to SDK — `DefaultAzureCredential` resolves the correct tenant |

# Quick Start: Hosted Foundry Agent

Opinionated happy-path for first-time users creating their first hosted Foundry agent. Safe defaults, minimal decisions.

> **Scope:** Defaults below are applied automatically when the user is silent. The user may override the language or sample explicitly; new-vs-existing Foundry project is handled inline. For anything not covered here, stop and read [create-hosted.md](create-hosted.md).

## When to Use This Skill

Use this when the request is to create a new hosted Foundry agent end-to-end — scaffold, provision, deploy, and smoke-test. Common overrides (language, region, sample, topic, existing project, existing model) are fine; bounce to [create-hosted.md](create-hosted.md) for anything else.

## Quick Reference

| Property | Default (when user is silent) | Override |
|----------|-------------------------------|----------|
| Language / runtime | Python 3.13 (`python_3_13`) | Any of `python_3_13`, `python_3_14`, `dotnet_10` |
| Sample | Featured basic starter for the chosen language (`azd ai agent sample list --featured-only --language <lang> --output json`) | User may name a different featured sample |
| Subscription | `az account show` | User may supply |
| Region | `northcentralus` | Ask user to confirm or pick another |
| Foundry project | Ask if the user doesn't mention one | create new → no `--project-id`; existing → pass `--project-id` (ARM ID / endpoint); no mention → stop and ask (existing vs new) |
| Model deployment | Whatever the sample's manifest declares | If user supplies a deployment name, `azd env set AZURE_AI_MODEL_DEPLOYMENT_NAME` after init |
| Deploy mode | `code` (no Docker, no ACR build) | — |
| Stops at | Deployed agent + remote smoke invoke + eval generation submitted | — |

## Workflow

Walk through every step in order. **Before Step 2**, scan the user's original prompt for any of these values: project name, language, subscription, region, existing Foundry project endpoint or ARM ID, existing model deployment name, agent topic/purpose. **Do not ask** for anything already supplied.

### Step 1 — Verify the environment

Two pre-flight checks — run each script and act on its `[OK]` / `[WARN]` / `[ACTION]` summary prefixes.

**1a — Canvas-first entry (GitHub Copilot app).** Detects whether the runtime is the GitHub Copilot app (`AI_AGENT=github_copilot_app_agent`) and the Foundry Agent Canvas extension is installed. If both are true, the canvas must be opened first so the user can authenticate and select a Foundry project before scaffolding. Run this check first (it can short-circuit the rest). **Skip 1a** if the user opts out, e.g. "skip the canvas" / "use the CLI".

```bash
./scripts/check-canvas-entry.sh     # macOS / Linux
./scripts/check-canvas-entry.ps1    # Windows (pwsh)
```

- **No `[ACTION]`** (only `[OK]`/`[WARN]`) — the gate doesn't apply (not in the Copilot app, or the canvas isn't installed). Continue to 1b.
- **`[ACTION]`** — the gate applies. If the user message's `<canvas-context>` already lists **`canvas="agent-builder"`**, the user has already driven the canvas — continue to 1b using their prompt/config (loop guard, independent of prompt wording). Otherwise `open_canvas` (`canvasId: "agent-builder"`), ask the user to **create the agent from the canvas: sign in, select a subscription + Foundry project, then Send**, then **stop — do not run 1b or scaffold**.

**1b — Tooling & auth.** Run the bundled script:

```bash
./scripts/verify-environment.sh     # macOS / Linux
./scripts/verify-environment.ps1    # Windows (pwsh)
```

Act on the summary prefixes:

- `[OK]` -- nothing to do.
- `[WARN]` -- non-blocking; continue.
- `[ACTION]` -- resolve first, then rerun the script. If `az` or `azd` is missing, ask before installing in interactive mode; install directly in non-interactive mode. For how to install `azd`, see <https://learn.microsoft.com/en-us/azure/developer/azure-developer-cli/install-azd>. In any mode, never run `az login` or `azd auth login`; stop and ask the user to log in manually before any init, provision, or deploy command. Missing `azure.ai.agents` / `azure.ai.projects` extensions may be resolved with `azd extension install <name>`.

### Step 2 — Collect remaining inputs (one batch)

For any values **not** already in the prompt, ask the rest in a single `AskUserQuestion` round:

| Value | Default | Notes |
|-------|---------|-------|
| Project / agent name | `ai-agent-<random6>` (6 lowercase alphanumeric chars) | Used as agent name, service key, and project directory. |
| Language | `python_3_13` | One of `python_3_13`, `python_3_14`, `dotnet_10`. |
| Subscription | `az account show --query id -o tsv` | Must be a GUID. |
| Region | `northcentralus` | Confirm or override. |
| Foundry project | Ask if the user doesn't mention one | User said create new → create a new one (no `--project-id`). User gave an existing project → use its ARM resource ID *or* Foundry project endpoint URL. User didn't mention a project at all → stop and ask, offering existing vs new. |
| Existing model deployment? | No (use sample manifest's model) | If Yes: collect the deployment name. |

If the user supplied only a **Foundry project endpoint** (not an ARM ID), resolve the ARM ID before Step 6:

```bash
./scripts/resolve-project-id.sh --endpoint "<foundry-project-endpoint>"     # macOS / Linux
./scripts/resolve-project-id.ps1 -Endpoint "<foundry-project-endpoint>"     # Windows (pwsh)
```

Use the returned `id` value. Never guess or construct the ARM ID from the endpoint.

### Step 3 — Pick the sample

```bash
azd ai agent sample list --featured-only --language <lang> --output json
```

> `--language` here takes the short form (`python`, `dotnetCsharp`) — not the runtime token (`python_3_13` fails with `unknown language`). The runtime tokens are only used in Step 6's `azd ai agent init --runtime ...`.

Pick the basic starter (e.g. `azd-ai-starter-basic` for Python — avoid samples with `parameters:` blocks requiring secrets). Capture the `manifestUrl`.

Step 6 needs `--runtime` and `--entry-point` values. These are CLI args, **not** fields in the manifest — use these standard defaults for the chosen language:

| Language | `--runtime` | `--entry-point` |
|----------|-------------|-----------------|
| Python | `python_3_13` | `main.py` |
| .NET | `dotnet_10` | `MyAgent.dll` |

### Step 4 — Create the project directory

```bash
mkdir <project-name>
cd <project-name>
```

### Step 5 — Pre-bootstrap with core `azd init`

This step writes `AZURE_SUBSCRIPTION_ID` + `AZURE_LOCATION` into the azd env *before* `azd ai agent init` runs, which prevents init from deferring model resolution and leaving the `{{AZURE_AI_MODEL_DEPLOYMENT_NAME}}` placeholder in the agent service's `environmentVariables`.

> `azd init` requires an **empty** directory — `--no-prompt` does **not** bypass the overwrite prompt and exits non-zero if files already exist. Step 4 created a fresh directory, so this is satisfied.

```bash
azd init -t Azure-Samples/azd-ai-starter-basic . \
  -e <project>-<random6> \
  --subscription <id> \
  -l <region> \
  --no-prompt
```

Use env name `<project>-<random6>` as the **default** to avoid collisions with stuck "Deleting"-state resource groups from prior runs. Use bare `<project>` only when you're confident the name has never been used in this subscription.

### Step 6 — Scaffold the agent

```bash
azd ai agent init --no-prompt \
  -m "<manifestUrl>" \
  --deploy-mode code \
  --runtime python_3_13 \
  --entry-point main.py \
  --agent-name <project>
```

Values you **must** substitute from Step 3 — do not pass placeholders or guesses:

- `--runtime`: exactly one of `python_3_13`, `python_3_14`, `dotnet_10` (the bare value `python` fails with `--runtime must be one of: python_3_13, python_3_14, dotnet_10`).
- `--entry-point`: the entry-point file the sample declares (e.g. `main.py`, not `app.py` — a wrong value scaffolds correctly but breaks local run and deploy).

If using an existing Foundry project, add `--project-id "<arm-id>"`.

⏳ May take time — init resolves the model catalog server-side. Wait for the prompt to return; do not interrupt.

`init` writes `azure.yaml` (appending the agent service), `src/<project>/.agentignore`, and the sample source files under `src/<project>/`.

### Step 7 — Customize the scaffolded sample (per user's original intent)

The scaffold is a generic working sample. Edit only what the user's original prompt asked for — touch tools, dependencies, or model config only when the user explicitly asked for external actions, APIs, tools, connectors, data lookup, or a specific model.

Typical changes:

- The agent service's `description:` in `azure.yaml` — update it to match the user's intent (this also feeds Step 15 eval generation).
- `src/<project>/<entry-point>` — update the system prompt / instructions to match the user's intent.

Only when the user explicitly asked for it:

- Add or modify tool / function-calling code in `<entry-point>`.
- Add dependencies to `pyproject.toml` / `requirements.txt` (Python) or `*.csproj` (.NET).
- Change the model in `azure.yaml services.ai-project.deployments[]` before Step 10 provision.

If the user's original prompt was generic (no specific agent purpose described), skip customization and ship the sample as-is.

### Step 8 — Write the agent instruction file (required)

Do **not** skip this. Add a one-line marker to the project's `AGENTS.md` (or `CLAUDE.md` in Claude Code) so future prompts reload this skill. If the file already exists, append the line; don't overwrite it or add a second file.

```
This project was built with the microsoft-foundry skill. Before working on or answering questions about foundry agents, read the microsoft-foundry skill first.
```

### Step 9 — Sanity-check the scaffold

Verify all four before continuing. If any check fails, pick **one** recovery path, then re-verify:

| Check | Expected | If failed |
|-------|----------|-----------|
| `azure.yaml services.ai-project.deployments[]` | Non-empty array with `name`, `model.{name,format,version}`, `sku.{name,capacity}` | Model resolution deferred — use recovery |
| Agent service `environmentVariables` `AZURE_AI_MODEL_DEPLOYMENT_NAME` (in `azure.yaml`) | Literal name **or** `${AZURE_AI_MODEL_DEPLOYMENT_NAME}` substitution | If literal `{{AZURE_AI_MODEL_DEPLOYMENT_NAME}}` (double braces): use recovery |
| Agent service `codeConfiguration.entryPoint:` (in `azure.yaml`) | Matches a real file in `src/<project>/` (e.g. `main.py` and `main.py` exists) | If mismatch (e.g. `entryPoint: app.py` but only `main.py` exists): edit `azure.yaml` to the real filename, then re-verify. Most often caused by passing a wrong `--entry-point` in Step 6. |
| `azure.yaml services:` keys | Only one `<project>` entry | If `<project>-2` exists: init was re-run; use recovery |

**Recovery paths** (pick based on whether Step 7 has already customized `src/<project>/`):

1. **Hand-fix in place** *(use when Step 7 customization is already done — preserves user code)* — edit `azure.yaml services.ai-project.deployments[]` to add the model block, replace `{{AZURE_AI_MODEL_DEPLOYMENT_NAME}}` in the agent service's `environmentVariables` with `${AZURE_AI_MODEL_DEPLOYMENT_NAME}`, then `azd env set AZURE_AI_MODEL_DEPLOYMENT_NAME <deployment-name>`.
2. **Clean re-init** *(use only when Step 7 has not run yet — destructive: deletes `src/<project>/`)* — delete `src/<project>/`, remove the `services.<project>:` block from `azure.yaml`, re-run Step 6.
3. **Interactive overwrite** *(loses Step 7 edits — re-resolves the model from the original manifest)* — re-run Step 6 *without* `--no-prompt`. When the collision prompt appears, **arrow-up to "Overwrite existing"** (default is *not* overwrite).

Never `azd env set AI_PROJECT_DEPLOYMENTS '[...]'` (single-escaped JSON breaks Bicep parse). Never `az cognitiveservices account deployment create` against this account (creates the deployment outside the azd lifecycle).

If recovery still fails → escape to [create-hosted.md](create-hosted.md).

### Step 10 — Provision Azure resources

> 🚦 **Project-selection gate (align with Step 2).** Only `azd provision` a new project when the user asked to create one. If the user gave an existing project, skip provision and use it. If the user didn't mention a project at all, stop and ask first — don't silently provision a new one.

```bash
azd provision --no-state --no-prompt
```

`--no-state` skips the existing-deployment check; safe here because the golden path starts from a fresh environment (Step 5). Keep it for this quickstart; you can omit it later when re-provisioning the same environment.

⏳ May take time — creates the resource group, Foundry account + project, model deployment, App Insights, Log Analytics. Wait for the prompt to return; do not interrupt.

### Step 11 — Wire local env vars

```bash
azd env get-values
```

Capture `FOUNDRY_PROJECT_ENDPOINT` and `AZURE_AI_MODEL_DEPLOYMENT_NAME`. Write `src/<project>/.env`:

```env
FOUNDRY_PROJECT_ENDPOINT=https://<account>.services.ai.azure.com/api/projects/<project>
AZURE_AI_MODEL_DEPLOYMENT_NAME=<deployment-name>
```

Also mirror them into the azd env (so `azd ai agent run` injects the right values — it reads azd env *before* `.env`):

```bash
azd env set AZURE_AI_PROJECT_ENDPOINT "<endpoint>"
azd env set AZURE_AI_MODEL_DEPLOYMENT_NAME "<deployment-name>"
```

### Step 12 — Local smoke test

Set up a venv with `uv` installed first. `azd ai agent run` installs Python dependencies on first start; with an activated venv that has `uv` available, it uses `uv` (seconds) instead of plain `pip` (minutes).

> **Important:** the venv must live in `src/<project>/` (next to `requirements.txt`). `azd ai agent run` resolves the venv relative to the service source directory; a venv at the project root is ignored and azd silently creates a second one without `uv`, wasting the speedup.

**Python:**
```bash
cd src/<project>
python -m venv .venv
# Activate the venv — pick the line for your shell:
.\.venv\Scripts\Activate.ps1                    # Windows pwsh
source .venv/bin/activate                       # macOS / Linux
python -m pip install uv
cd -                                             # back to project root for the azd commands below
```

**.NET:** no pre-install step — `azd ai agent run` runs `dotnet restore` itself on first start.

Run the agent locally. For Python, do this **with the service-dir venv still activated** — activation is what lets `azd ai agent run` find `uv` for the fast dependency install. `azd ai agent run` **is** the local server — a foreground process holding port 8088 that must stay alive from start, through every `invoke --local`, until you explicitly stop it.

Start it in a **managed** background session your shell tool can poll and stop (most tools detect a long-running foreground process and return a session/shell id — use that id). Do **not** use job operators (`bash &`, `nohup`, `start /B`, popped windows): on Linux/macOS the child gets `SIGHUP` and **dies when its parent bash exits**, so the next command sees `could not connect` even though `ss` from inside the *same* bash just showed `:8088` bound.

> ⚠️ **Readiness gate — do not skip.** After starting `azd ai agent run`, **watch the server log for the ready line, something like `Running` (e.g. `Running on http://0.0.0.0:8088`) — not just `Starting …`**, which azd prints as a banner before the Python process has bound the socket. Invoking before the socket is bound fails with `could not connect`.
> - **Never invoke before the most recent log read shows the ready line.** Premature invokes waste a poll cycle and return a misleading `could not connect`.
> - **Poll short — 2–5s per read.** Boot time is unbounded; long sleeps cost wall-clock directly. No 15s+ blocks or `sleep N` waits.
> - **Don't substitute log polling** with `sleep N && curl`, `netstat` / `ss` / `lsof`, or `ps aux` probes — only the log tells you readiness.
> - **If `invoke --local` fails,** re-read the server log. Error before the ready line (missing env var, auth, port in use) → fix the cause and restart `azd ai agent run` in the managed session. Ready line present but request still fails → the issue is in the request, not the server. Either way, do **not** bypass with `python main.py` or raw `curl POST /responses` — those skip the wiring the deployed agent uses.
> - **If `invoke --local` returns `could not connect` after you saw the ready line in a previous shell,** the server died when that shell exited (classic `&` symptom). Restart in the managed session — do not retry with another `&`.

```bash
azd ai agent run --no-inspector
```

Smoke-invoke (local):

```bash
azd ai agent invoke --local "<short representative prompt for the agent's purpose>"
```

Stop the local server via the managed session's stop primitive before continuing — a lingering process holds files in the project and breaks later cleanup.

### Step 13 — Deploy

```bash
azd deploy --no-prompt
```

⏳ May take time — zips `src/<project>/` (respecting `.agentignore`), uploads to Foundry, builds runtime remotely, registers agent version. Wait for the prompt to return; do not interrupt.

### Step 14 — Verify + remote smoke

```bash
azd ai agent show --output json
```

Expect `"status": "active"` (or `"deployed"`) and an `agent_endpoints` map.

Remote invoke (billed):

```bash
azd ai agent invoke "<short representative prompt>"
```

`azd ai agent invoke` has **no `--force` flag**. If the command succeeds, read the response. If it surfaces a confirmation prompt or message, summarize the cost implication for the user (*"this will call the deployed agent and incur model usage charges"*), get explicit consent, and re-run — do **not** invent flags.

### Step 15 — Submit eval suite generation (async, fire-and-forget)

> ⚠️ **Pre-summary gate.** Do not write the Step 16 final summary until this step has been submitted. The eval suite is part of the deployment artifact; skipping it ships an incomplete result.

Read the agent service's `description:` from `azure.yaml` (the value you set in Step 7) and pass it as `--gen-instruction`:

```bash
azd ai agent eval generate --gen-instruction "<agent service description>" --no-wait --no-prompt
```

Expected output:

```
Eval generate submitted (async)
   dataset generation: datagen-<id> (queued)
   evaluator generation: evaluatorgen-<id> (in_progress)
   Config written to: src/<project>/eval.yaml
   When ready, run:
     azd ai agent eval run
```

Generation runs server-side and takes several minutes. Tell the user:

> *"Eval suite generation submitted. Run `azd ai agent eval run` whenever you're ready — it'll wait for generation to finish and execute the eval in one step."*

### Step 16 — Final summary

Produce a concise summary covering: agent name/version/status/endpoints, a Playground link, the resources created, and the three follow-up commands below. Read `playground_url` directly from `azd ai agent show --output json`. If it is absent, construct the Playground URL from `azd env get-values`:

```
https://ai.azure.com/nextgen/r/{encodedSubId},{resourceGroup},,{accountName},{projectName}/build/agents/{agentName}/build?version={agentVersion}
```

`encodedSubId` = URL-safe base64 of the subscription GUID, padding stripped:

```bash
python -c "import base64,uuid;print(base64.urlsafe_b64encode(uuid.UUID('<SUBSCRIPTION_ID>').bytes).rstrip(b'=').decode())"
```

Three follow-up commands to include:

```bash
azd ai agent invoke "<follow-up message>"   # chat with the deployed agent (billed)
azd ai agent eval run                       # finalize + run the eval suite (Step 15)
azd down                                    # tear down all resources when done
```

## Error Handling

| Symptom | Fix |
|---------|-----|
| `azd ai agent init` fails with `--runtime must be one of: python_3_13, python_3_14, dotnet_10` | You passed a bare value like `python`. Use the full runtime token (e.g. `python_3_13`). |
| `azd ai agent init` fails with `--entry-point is required when using --deploy-mode code with --no-prompt` | Pass `--entry-point <filename>` matching the entry-point file the sample declares (from Step 3). |
| `codeConfiguration.entryPoint` doesn't match any file in `src/<project>/` | You guessed the entry-point in Step 6. Edit the agent service in `azure.yaml` to the real filename (verify with `ls src/<project>/`). No re-init needed. |
| `azd deploy` postdeploy hook fails with missing `AZURE_TENANT_ID` | Run `az account show --query tenantId -o tsv` and `azd env set AZURE_TENANT_ID <tenant-id>`, then re-run `azd deploy --no-prompt`. The deployed agent version from the first deploy is still valid; the postdeploy hook just registers env vars. |
| Scaffold sanity check fails (Step 9) | Pick a recovery path from Step 9. If still failing → [create-hosted.md](create-hosted.md). |
| Local invoke returns model `404` / wrong deployment | Stale `AZURE_AI_MODEL_DEPLOYMENT_NAME` in azd env overrides `.env`. Re-run Step 11 to sync both. |
| `azd ai agent invoke ... --force` returns `unknown flag: --force` | `--force` is not a valid flag for invoke. Re-run without it. |
| Anything else | Escape to [create-hosted.md](create-hosted.md). |

## Escape Hatch

If any step fails in a way not covered above, the output looks unexpected, or the user's request drifts outside what this quickstart covers → **stop improvising**. Read [create-hosted.md](create-hosted.md) and follow its full workflow.

# Agent Tools

This file is the **index** for every tool an agent can use. For each tool, it points to a dedicated reference file, and — where the tool is also available through a [toolbox](use-toolbox-in-hosted-agent.md) — lists the toolbox `type` value.

Two delivery paths exist:

- **Prompt agent** — the agent definition declares tool classes directly (`CodeInterpreterTool`, `MCPTool`, …). Use the SDK class column and the per-tool reference.
- **Hosted agent via toolbox** — the agent connects to a single MCP endpoint that exposes all tools declared in a toolbox version. Use the `type` column and see [use-toolbox-in-hosted-agent.md](use-toolbox-in-hosted-agent.md). For wiring the underlying project connection (catalog tile or generic remote MCP), see [foundry-tool-catalog.md](foundry-tool-catalog.md).

> 💡 **Authoritative tool shapes:** the source-of-truth for every tool's wire shape is the **Foundry Agents typespec** on the `main` branch of [`Azure/azure-rest-api-specs`](https://github.com/Azure/azure-rest-api-specs/tree/main/specification/cognitiveservices). When in doubt about a field name, default, or new tool type that isn't yet documented here, load the typespec directly — it's updated as tools are added/changed.

## Tool Summary

| Tool | Prompt-agent SDK class | Toolbox `type` | Connection? | Reference |
|------|------------------------|----------------|-------------|-----------|
| Code Interpreter | `CodeInterpreterTool` | `code_interpreter` | No | [tool-code-interpreter.md](tool-code-interpreter.md) |
| Function calling (client-side) | `FunctionTool` | — (client-side only) | No | [tool-function-calling.md](tool-function-calling.md) |
| File Search | `FileSearchTool` | `file_search` | No (vector store required) | [tool-file-search.md](tool-file-search.md) |
| Web Search (preview) | `WebSearchPreviewTool` | `web_search` (with optional `web_search.custom_search_configuration` for Bing Custom Search) | No (basic Bing); **Yes** for Grounding with Bing Custom Search — the connection scopes grounding to specific domains | [tool-web-search.md](tool-web-search.md) |
| Bing Grounding | `BingGroundingAgentTool` | — (N/A in toolbox; the toolbox path uses `web_search` with `web_search.custom_search_configuration`) | Yes (Bing) — prompt-agent path only | [tool-bing-grounding.md](tool-bing-grounding.md) |
| Azure AI Search | `AzureAISearchAgentTool` | `azure_ai_search` | Yes (Search) | [tool-azure-ai-search.md](tool-azure-ai-search.md) |
| MCP server (remote) | `MCPTool` | `mcp` | Optional (none / static key / project MI / OAuth) | [tool-mcp.md](tool-mcp.md); toolbox attach via [foundry-tool-catalog.md](foundry-tool-catalog.md) |
| OpenAPI tool | (n/a as a single class) | `openapi` | Conditional — `connection` auth requires `project_connection_id`; **`managed_identity` auth does NOT** (the project MI is used directly with an `audience`) | [tool-openapi.md](tool-openapi.md) |
| Agent-to-Agent (A2A) | (n/a as a single class) | `a2a_preview` | Optional | [tool-a2a.md](tool-a2a.md) |
| Agent Memory | `MemorySearchTool` | — (separate memory store) | Yes (project MI + embedding model) | [tool-memory.md](tool-memory.md) |
| **Work IQ (preview)** | (n/a — server-side only) | `work_iq_preview` | Yes (Work IQ BYO-Entra-app OAuth connection) | [tool-work-iq.md](tool-work-iq.md) |
| **Fabric IQ (preview)** | (n/a — server-side only) | `fabric_iq_preview` | Yes (Fabric IQ Entra-app OAuth or managed-OAuth connection) | [tool-fabric-iq.md](tool-fabric-iq.md) |
| **Tool Search (preview)** | (n/a — toolbox-side configuration directive) | `toolbox_search_preview` | No | [tool-tool-search.md](tool-tool-search.md) |

> ⚠️ **Default for web search:** Use `WebSearchPreviewTool` (`type: web_search`) unless the user explicitly requests Bing Grounding or Bing Custom Search.

> Combine multiple tools on one agent or one toolbox version. The model decides which to invoke. For multi-tool toolbox limits (at most one unnamed tool per type, unique `server_label` per MCP tool) see [toolbox-reference.md](toolbox-reference.md#multi-tool-toolbox-constraint).

## How to use this index

When you need details for a specific tool, **load that tool's reference file directly** — each one is self-contained (shape, requirements, references). Don't try to keep all tools in context at once.

For the toolbox runtime contract (endpoint, auth, MCP protocol, citation patterns, troubleshooting) see [toolbox-reference.md](toolbox-reference.md). For wiring a toolbox into a hosted agent (env vars, samples, tracing) see [use-toolbox-in-hosted-agent.md](use-toolbox-in-hosted-agent.md).

## Adjacent (not a `type` in a toolbox version)

- **Agent Memory** — use the `MemorySearchTool` SDK class on prompt agents; for hosted agents, configure the memory store via the project (separate from the toolbox). See [tool-memory.md](tool-memory.md).
- **Routines (preview)** — not a tool; an agent **trigger** (`schedule` / `timer` / `github_issue` / `custom`) that invokes an existing agent. Event-based routines are powered by the same **Connector Namespace** that backs catalog-MCP / managed-MCP connectors. See the [public Routines docs](https://learn.microsoft.com/azure/foundry/agents/how-to/use-routines).

## References

- **[Foundry Agents typespec (`main`)](https://github.com/Azure/azure-rest-api-specs/tree/main/specification/cognitiveservices)** — authoritative tool shapes
- [Tool Catalog](https://learn.microsoft.com/azure/foundry/agents/concepts/tool-catalog)
- [Toolbox (preview)](https://learn.microsoft.com/azure/foundry/agents/how-to/tools/toolbox)
- [use-toolbox-in-hosted-agent.md](use-toolbox-in-hosted-agent.md) — wiring a toolbox into a hosted agent
- [toolbox-reference.md](toolbox-reference.md) — toolbox runtime contract
- [foundry-tool-catalog.md](foundry-tool-catalog.md) — project connections for remote tools

# Microsoft Agent Framework — Best Practices for Hosted Agents

Best practices when building hosted agents with Microsoft Agent Framework for deployment to Foundry Agent Service.

## Official Resources

| Resource | URL |
|----------|-----|
| **GitHub Repo** | https://github.com/microsoft/agent-framework |
| **MS Learn Overview** | https://learn.microsoft.com/agent-framework/overview/agent-framework-overview |
| **Quick Start** | https://learn.microsoft.com/agent-framework/tutorials/quick-start |
| **User Guide** | https://learn.microsoft.com/agent-framework/user-guide/overview |
| **Hosted Agents Concepts** | https://learn.microsoft.com/azure/ai-foundry/agents/concepts/hosted-agents |
| **Python Samples (MAF repo)** | https://github.com/microsoft/agent-framework/tree/main/python/samples |
| **.NET Samples (MAF repo)** | https://github.com/microsoft/agent-framework/tree/main/dotnet/samples |
| **PyPI** | https://pypi.org/project/agent-framework/ |
| **NuGet** | https://www.nuget.org/profiles/MicrosoftAgentFramework/ |

## Installation

**Python:** `pip install agent-framework agent-framework-foundry-hosting` (installs all sub-packages)

**.NET:** `dotnet add package Microsoft.Agents.AI`

## Hosting Adapter

Hosted agents must expose an HTTP server using the hosting adapter. This enables local testing and Foundry deployment with the same code.

**Python adapter packages:** `agent_framework_foundry_hosting`

**.NET adapter packages:** `Azure.AI.AgentServer.Core`, `Microsoft.Agents.AI.Foundry.Hosting`

The adapter handles protocol translation between Foundry request/response formats and your framework's native data structures, including conversation management, message serialization, and streaming.

> 💡 **Tip:** Make HTTP server mode the default entrypoint (no flags needed). This simplifies both local debugging and containerized deployment.

## Key Patterns

### Python: Credentials

For **local development**, use `DefaultAzureCredential` from `azure.identity`. In production, use `ManagedIdentityCredential`. See [auth-best-practices.md](../../../references/auth-best-practices.md).

### Python: Environment Variables

Always use `load_dotenv(override=False)` so environment variables set by Foundry at runtime take precedence over local `.env` values.

Required `.env` variables:
- `FOUNDRY_PROJECT_ENDPOINT` — project endpoint URL
- `FOUNDRY_MODEL_DEPLOYMENT_NAME` — model deployment name

### Authentication

If explicitly asked to use API key instead of managed identity, then use AzureOpenAIResponsesClient and pass in api_key parameter to it.

### Agent Naming Rules

Agent names must: start/end with alphanumeric characters, may contain hyphens in the middle, max 63 characters. Examples: `MyAgent`, `agent-1`. Invalid: `-agent`, `agent-`, `sample_agent`.

### Python: Virtual Environment

Always use a virtual environment. Never use bare `python` or `pip` — use venv-activated versions or full paths (e.g., `.venv/bin/pip`).

## Workflow Patterns

Agent Framework supports single-agent and multi-agent workflow patterns using graph-based orchestration:

- **Single Agent** — Basic agent with tools, RAG, or MCP integration
- **Multi-Agent Workflow** — Graph-based orchestration connecting multiple agents and deterministic functions
- **Advanced Patterns** — Reflection, switch-case, fan-out/fan-in, loop, human-in-the-loop

For workflow samples and advanced patterns, search the [Agent Framework GitHub repo](https://github.com/microsoft/agent-framework).

## Debugging

Use [Foundry Toolkit for VS Code (Formerly AI Toolkit)](https://marketplace.visualstudio.com/items?itemName=ms-windows-ai-studio.windows-ai-studio) with the `agentdev` CLI tool for interactive debugging:

1. Install `debugpy` for VS Code Python Debugger support
2. Install `agent-dev-cli` (pre-release) for the `agentdev` command
3. Key debug tasks: `agentdev run <entrypoint>.py --port 8087` starts the agent HTTP server, `debugpy --listen 127.0.0.1:5679` attaches the debugger, and the `ai-mlstudio.openTestTool` VS Code command opens the Agent Inspector UI

For VS Code `launch.json` and `tasks.json` configuration templates, see [Foundry Toolkit Agent Inspector — Configure debugging manually](https://github.com/microsoft/vscode-ai-toolkit/blob/main/doc/agent-test-tool.md#configure-debugging-manually).

## Common Errors

| Error | Cause | Fix |
|-------|-------|-----|
| `ModuleNotFoundError` | Missing SDK | `pip install agent-framework agent-framework-foundry-hosting` in venv |
| Credential error | Wrong import | Use `azure.identity.DefaultAzureCredential` (local dev) or `ManagedIdentityCredential` (production) |
| Agent name validation error | Invalid characters | Use alphanumeric + hyphens, start/end alphanumeric, max 63 chars |
| Hosting adapter not found | Missing package | Install `agent-framework-foundry-hosting` |

# azd ai CLI Reference

Core mental model for the `azd ai agent` extension. Use this when you need to understand command surface, file layout, or where a given setting lives.

## CLI surface

```bash
azd ai project show                  # which Foundry project endpoint is active
azd ai agent show                    # is the agent deployed? what version?
azd ai agent doctor                  # full health check, suggests fixes

azd ai agent sample list             # curated catalog -- pick a manifestUrl
azd ai agent init -m <manifestUrl>   # scaffold from a sample
azd ai agent init --src <dir>        # scaffold from existing source

azd ai agent run                     # start the agent on localhost:8088
azd ai agent invoke "<msg>"          # remote invoke (billed; gated)
azd ai agent invoke --local "<msg>"  # local invoke (no billing)

azd provision                        # core azd; creates Foundry project + infra
azd deploy                           # core azd; packages + registers new agent version
azd ai agent endpoint update         # patch agentEndpoint / agentCard in place

azd ai connection list / show / create / update / delete
azd ai toolbox list / show / create / publish / delete
azd ai toolbox connection add / remove / list
azd ai toolbox versions list

azd ai agent files list / show / upload / download / delete / stat / mkdir
azd ai agent sessions list / show / create / update / delete
azd ai agent monitor                 # per-session log stream (SSE)

azd ai agent eval generate / run / show / update / list
azd ai agent optimize / optimize status / optimize apply / optimize deploy / optimize cancel
```

Read-only commands accept `--output json` and never require `--force`. Write commands are gated by a confirmation envelope (see "Confirmation envelope" below).

## The azure.yaml service block

After `azd ai agent init`, every hosted agent is defined as a **service block in `azure.yaml`** (`host: azure.ai.agent`) plus the active azd env; init consolidates the sample's definition into `azure.yaml`.

| Location | What it holds |
|------|---------------|
| `azure.yaml services.<name>` (the agent) | `host: azure.ai.agent`, `kind`, `name`, `project`, `language`, `uses`, `protocols`, `environmentVariables`, `codeConfiguration` / `docker` / `image`, `container.resources`, `description`, `agentEndpoint`, `agentCard`, `startupCommand`. |
| `azure.yaml services.ai-project` | Model `deployments[]` (`host: azure.ai.project`). The agent links to it via `uses: [ai-project]`. |
| `.azure/<env>/.env` (`azd env set`) | Secrets and `PARAM_<CONN>_<KEY>` credential values referenced from `azure.yaml`. |

`azd deploy` reads the agent service block and creates a new immutable agent version. `azd provision` reads `services.ai-project.deployments[]` (and any connection/toolbox services) and applies them via Bicep.

`agent.manifest.yaml` (the file passed to `-m`) is the seed format -- it is NOT on disk after init. Init folds its `parameters:` / `resources:` blocks into the `azure.yaml` service block and the azd env.

> **Local vs API field names.** Local `azure.yaml` uses **camelCase** (`codeConfiguration`, `entryPoint`, `dependencyResolution`, `environmentVariables`). The deployed definition returned by `azd ai agent show` / the Foundry `agent_get` API uses **snake_case** (`code_configuration`, `entry_point` as an array, `environment_variables`). Don't mix the two.

### Hosted agent service block (code deploy)

```yaml
services:
  ai-project:
    host: azure.ai.project
    deployments:
      - name: gpt-4.1-mini
        model:
          format: OpenAI
          name: gpt-4.1-mini
          version: "2024-04-09"
        sku:
          name: GlobalStandard
          capacity: 50
  my-agent:
    project: src/my-agent
    host: azure.ai.agent
    language: python
    uses:
      - ai-project
    kind: hosted
    name: my-agent
    description: A hosted agent.
    codeConfiguration:
      runtime: python_3_13
      entryPoint: main.py
      dependencyResolution: remote_build   # or "bundled"
    container:
      resources:
        cpu: "0.5"
        memory: 1Gi
    environmentVariables:
      - name: AZURE_AI_MODEL_DEPLOYMENT_NAME
        value: ${AZURE_AI_MODEL_DEPLOYMENT_NAME}
    protocols:
      - protocol: responses
        version: 1.0.0
```

- `protocols` -- `responses` (OpenAI), `invocations` (A2A), `invocations_ws`. Editing requires `azd deploy`.
- `container.resources` -- valid tiers: `0.25/0.5Gi`, `1/2Gi`, `2/4Gi`.
- `environmentVariables` -- `${VAR}` resolves from the active azd env. Not for secrets.
- `codeConfiguration` present -> direct code deploy (ZIP, Foundry builds). Absent -> container/ACR deploy: the service uses `language: docker` + `docker.remoteBuild: true` + `startupCommand` (and `image:` skips the Dockerfile build).
- In non-interactive mode, `azd ai agent init` defaults to container deploy. Pass `--deploy-mode code --runtime <runtime> --entry-point <file>` during init to get `codeConfiguration`.
- `agentEndpoint` / `agentCard` -- patch in place with `azd ai agent endpoint update` (no new version).
- `deployments[]` (under the `ai-project` service) -- model deployments provisioned via Bicep. `name` is the literal Azure deployment resource name the agent references through `AZURE_AI_MODEL_DEPLOYMENT_NAME`.
- Connections/toolboxes -- created with `azd ai connection` / `azd ai toolbox` and consumed via a `TOOLBOX_<NAME>_MCP_ENDPOINT` env var (see [tools](tools.md)). The emerging declarative form models them as top-level `azure.ai.connection` / `azure.ai.toolbox` services linked via `uses:`.

## State (azd env vars)

| Variable | Read by | Where to set |
|----------|---------|--------------|
| `AZURE_AI_PROJECT_ENDPOINT` | Every `azd ai agent` command | `azd env set` or `azd ai project show` |
| `AZURE_AI_PROJECT_ID` | `azd ai agent show` (playground URL) | `azd env set` |
| `AZURE_SUBSCRIPTION_ID`, `AZURE_LOCATION` | `azd provision` | `azd init --subscription/-l` (or `azd config set defaults.subscription/location`) |
| `AGENT_<SVC>_NAME` / `_VERSION` / `_<PROTO>_ENDPOINT` | Auto-written by deploy | Auto |
| `PARAM_<CONN>_<KEY>` | Connection credentials in `azure.yaml` | `azd env set` |

Manage with `azd env get-values`, `azd env set`, `azd env list`, `azd env new`, `azd env select`.

The platform also injects `FOUNDRY_*` and `AGENT_*` into the running container at runtime. **Never** put these in the agent service's `environmentVariables` section.

## Resolving subscription / location

`azd ai project show` returns only the Foundry project endpoint. For subscription / location, try in order:

1. `azd config get defaults`
2. `azd env get-values`
3. Ask the user.
4. Last resort, with explicit consent: `az account list --output json`.

For the Foundry project ARM ID (`--project-id`), ask the user: "New project, or use an existing one?" If existing, ask for the ID and hint where to find it (https://ai.azure.com -> Operate -> Admin). Do NOT shell out to `az cognitiveservices` -- it returns the wrong resource shape.

## Common error codes

- `not_logged_in` / `login_expired` -- ask the user to run `azd auth login`.
- `missing_project_endpoint` -- run `azd provision`, or `azd env set AZURE_AI_PROJECT_ENDPOINT <url>`.
- `project_not_found` -- cwd has no `azure.yaml`. Move to project root or run init.
- `invalid_agent_manifest` -- the agent service block is malformed. Run `azd ai agent doctor` and read the named field.
- `invalid_connection` -- inspect with `azd ai connection show <name>`.
- `eval_config_invalid` -- `eval.yaml` failed validation. Run `azd ai agent doctor`.
- `agent_definition_not_found` -- deployed name doesn't match `azure.yaml`. Re-deploy from project root.

Any unfamiliar `code` value is safe to surface verbatim to the user.

# Foundry Tool Catalog — Project Connections for Remote Tools

Reference for wiring a **remote tool** (catalog tile or generic MCP server) into a Foundry project as a `RemoteTool` project connection, so a toolbox can attach to it.

> 🚦 **Toolbox creation gate:** before creating a toolbox/connection, you MUST read the boundary rules in [create-hosted.md → Toolbox creation boundary](../create-hosted.md#toolbox-creation-boundary) and follow them, then continue with the rest of this file.

Three catalog backends cooperate: the **asset-gallery** index discovers connectors, the Logic Apps **managedApis** GET supplies OAuth metadata, and the Logic Apps **apiOperations** GET supplies the operation list and input schemas. Skip these calls only for fully BYO `generic_mcp` servers — every catalog-MCP or connector-namespace flow needs all three.

> 📘 For the toolbox MCP endpoint, protocol, and testing, see [toolbox-reference.md](toolbox-reference.md).
> 📘 For prompt-agent MCP wiring (without a toolbox), see [tool-mcp.md](tool-mcp.md).

## When to use this reference

Use when the user mentions any of:

- *Build → Tools → Connect a tool* (any subtab — Configured, Catalog, Custom)
- "Tool connection", "Remote MCP", "Catalog tile", "Custom · Preview"
- A specific catalog tile (GitHub, Box, Pipedrive, monday.com, Microsoft Learn, …)
- `RemoteTool` connection, `gateway_connector`, `catalog_MCP`, `generic_mcp`
- **Connector Namespace** / managed MCP server (powered by the Connector Namespace)
- "Bring my own OAuth App" (BYO `client_id` + `client_secret`) for a catalog connector
- Discovering connector operations (`x-ms-operations` / Logic Apps `apiOperations`) or trigger support (`x-ms-trigger`) via the catalog APIs

Do **not** use for: non-tool connections (Azure OpenAI, AI Search account, Storage), or general toolbox CRUD beyond the attach-and-verify recipe below.

## Inputs to gather upfront

Before generating any PUT body, ask the user in one batched question for:

1. **Subscription id**
2. **Resource group**
3. **Cognitive Services account name** (the Foundry account)
4. **Project name** (under the account)
5. **Connection name** — lowercase, `[a-z0-9-]`, ≤ 24 chars (e.g. `box-1`, `gh-byo`)
6. **Tool scenario in plain language** — e.g. "list my files in Box", "create issues on GitHub". Map this onto operations from the connector's `apiOperations` catalog for `gateway_connector`, or onto the catalog MCP server's `tools/list` for `catalog_MCP` / BYO.
7. **Toolbox name** to attach into for verification (defaults to `default-tb`)
8. **Secrets** (BYO `clientId` / `clientSecret`, `CustomKeys` header value, …) — ask the user to **type these directly into the terminal**, never via tooling that echoes them

The caller's AAD `oid` / `tid` (needed only for the consent-link step) are auto-discovered via `az ad signed-in-user show --query id -o tsv` and `az account show --query tenantId -o tsv`. For a service-principal caller, use `az ad sp show --id <appId>` instead. These values can also be read from the `oid` / `tid` claims on the ARM bearer token; the gateway validates the caller principal owns them.

## ARM endpoint (shared by every variant)

```
PUT https://management.azure.com/subscriptions/{sub}/resourceGroups/{rg}
    /providers/Microsoft.CognitiveServices/accounts/{acct}
    /projects/{proj}/connections/{name}?api-version=2025-04-01-preview
```

### Preflight RBAC

Caller needs **Azure AI Developer** or **Cognitive Services Contributor** on the project scope. Run this before the first PUT to surface 403s early:

```pwsh
$oid = az ad signed-in-user show --query id -o tsv
$projId = "/subscriptions/$sub/resourceGroups/$rg/providers/Microsoft.CognitiveServices/accounts/$acct/projects/$proj"
az role assignment list --assignee $oid --scope $projId --all `
  --query "[?roleDefinitionName=='Azure AI Developer' || roleDefinitionName=='Cognitive Services Contributor'].roleDefinitionName" -o tsv
```

Empty output → caller lacks the required role; expect `403 AuthorizationFailed` on PUT until granted.

### Common request template

```pwsh
$tok = az account get-access-token --resource "https://management.azure.com" --query accessToken -o tsv
$h   = @{ Authorization = "Bearer $tok"; "Content-Type" = "application/json" }
$uri = "https://management.azure.com/subscriptions/$sub/resourceGroups/$rg/providers/Microsoft.CognitiveServices/accounts/$acct/projects/$proj/connections/${connName}?api-version=2025-04-01-preview"
Invoke-WebRequest -Method PUT -Headers $h -UseBasicParsing -Body $body -Uri $uri
```

### Body invariants

- `properties.target` is **required** for every `authType` (validation rejects empty). The exact value depends on the variant — see each body shape. For `gateway_connector` specifically, the literal string `"https://placeholder"` is the correct value on PUT #1 and is **rewritten by the platform on PUT #2** to the real gateway URL.
- `properties.group` is server-filled (`GenericProtocol` for `RemoteTool`).
- `properties.credentials` is scrubbed to `null` on GET.
- `properties.peRequirement` defaults to `"NotRequired"`.

Allowed `authType` for `category=RemoteTool` (per `api-version=2025-04-01-preview`):
`None, CustomKeys, OAuth2, ProjectManagedIdentity, DeveloperConnection, UserEntraToken, AgentUserImpersonation, AgenticIdentityToken, AgenticUser, UserTokenAndProjectManagedIdentity`. `ApiKey` is **rejected** for `RemoteTool`. The authoritative list is whatever the [Cognitive Services projects API reference](https://learn.microsoft.com/rest/api/aiservices/) returns for the current API version — if you hit `invalid_payload: unsupported authType`, re-check against the schema for the version you're calling.

## Decision tree

| User scenario | `authType` | `metadata.type` | Notes |
|---|---|---|---|
| Catalog tile tagged "Custom · Preview" (Box, Pipedrive, GitHub, Salesforce, Outlook, …) | `OAuth2` | `gateway_connector` | **Connector-namespace managed MCP.** Powered by the Connector Namespace in your Foundry account; the namespace handles OAuth, token storage, and per-user passthrough. Needs **two** PUTs plus `listConsentLinks` per caller (see [Gateway connector full flow](#gateway-connector-full-flow)). |
| Catalog MCP tile with Microsoft-managed OAuth (no `client_id` needed) | `OAuth2` | `catalog_MCP` | Foundry brokers the OAuth app for you. The Catalog API tile **prepopulates** `target` (server URL); `listConsentLinks` flow same as gateway. |
| Catalog MCP tile with **your own** OAuth App | `OAuth2` | (omit) | Supply your own `client_id` + `client_secret` + raw `authorizationUrl` / `tokenUrl` / `scopes`. Do **not** mix BYO `credentials` with `metadata.type=catalog_MCP`. See [BYO OAuth caveats](#byo-oauth-app-against-a-catalog-mcp-server). |
| Remote MCP, Azure-side identity (project MI calls the server) | `ProjectManagedIdentity` | `catalog_MCP` *(when listed)* or `generic_mcp` | For catalog-listed MCP servers, prefer `catalog_MCP` so `target` is prepopulated. Requires `audience` in `metadata`. See [PMI limitations](#projectmanagedidentity-limitations). |
| Remote MCP, static shared secret / header key | `CustomKeys` | `catalog_MCP` *(when listed)* or `generic_mcp` | Header **name and format** are NOT always `Authorization: Bearer ...`. Read the required header name from the Catalog API entry's `x-ms-connection-parameters` and use that exact name in `credentials.keys`. |
| Remote MCP, user's Entra token forwarded | `UserEntraToken` | `generic_mcp` | Per-user identity passthrough. Not supported when the agent is published to Teams. Pair with `metadata.audience` for the upstream resource URI. |
| Custom OpenAPI / A2A tool (no MCP) | varies | n/a | Use the Custom subtab shapes; outside the MCP toolbox path. See [Custom subtab — OpenAPI / A2A](#custom-subtab--openapi--a2a). |

## Catalog APIs — three backends, three calls

There are **three** read endpoints the portal hits to populate a connection form. Programmatic callers should use the same three.

### 1. Asset-gallery (Foundry's index)

```
POST https://eastus.api.azureml.ms/asset-gallery/v1.0/tools
Headers:
  Authorization: Bearer <token for https://management.azure.com>
  Content-Type: application/json
Body:
{
  "freeTextSearch": "*",
  "filters": [
    { "field": "entityContainerId", "operator": "eq",       "values": ["connectors-registry-prod-bl"] },
    { "field": "type",              "operator": "eq",       "values": ["tools"] },
    { "field": "annotations/name",  "operator": "contains", "values": ["<name>"] }
  ],
  "pageSize": 20
}
```

- **Catalog lives only in `eastus`.** `westus2.api.azureml.ms` returns `totalCount=0` for the same body. `entityId`s are portable across project regions.
- Use this **only to discover the connector's `entityId`** — pull `objectId` out of the returned `entityId` (e.g. `…/objectId/github`). That `objectId` is the `connectorName` you pass to PUTs and the next two catalog calls.
- The response is a **thin index**. `properties.remotes[]`, `xMsSecuritySchemes`, OAuth endpoints, scopes, and operation schemas are **not** included. Direct `GET /asset-gallery/v1.0/tools/{entityId}` returns 404. There is no expand/projection flag that surfaces these fields — fetch them from calls 2 and 3 below.

Two registries are indexed here — distinguished by `entityContainerId`:

| Registry | `entityContainerId` | Contents | Pair with |
|---|---|---|---|
| Public catalog | `connectors-registry-prod-bl` | Catalog connector definitions (GitHub, Box, Salesforce, …). | `metadata.type=catalog_MCP` or `gateway_connector` |
| Private MCP entries | `registry-prod-bl` | MCP-server entries used by the portal Connections UI (e.g. `github-mcp-server`). Sometimes carries a canonical MCP URL when the public-catalog row lacks `remotes[]`. | `metadata.type=catalog_MCP` |

Always query both when surfacing "available tools" to a user — the private MCP entries can fill gaps in the public catalog row.

### 2. Logic Apps **managedApis** — OAuth source-of-truth

```
GET https://management.azure.com/subscriptions/{sub}
    /providers/Microsoft.Web/locations/{region}/managedApis/{connectorName}
    ?api-version=2016-06-01
```

`connectorName` is the `objectId` from the asset-gallery `entityId`. Verified response shape for `github` (2026-05-21):

```jsonc
{
  "properties": {
    "displayName": "GitHub",
    "runtimeUrls": ["https://logic-apis-eastus.azure-apim.net/apim/github"],
    "connectionParameters": {
      "token": {
        "type": "oauthSetting",
        "oAuthSettings": {
          "identityProvider": "GitHub",
          "clientId": "faa5f56b825cbc649ae1",          // Microsoft's default OAuth-App id
          "scopes": ["repo","workflow","read:org","admin:org"],
          "redirectMode": "Direct",
          "redirectUrl": "https://logic-apis-eastus.consent.azure-apim.net/redirect"
        }
      }
    }
  }
}
```

**Raw `authorizationUrl` / `tokenUrl` are NOT in this response.** Logic Apps abstracts them via the `identityProvider` string and resolves them inside the gateway. For BYO you must map `identityProvider → endpoints` yourself. Known mappings:

| `identityProvider` | `authorizationUrl` | `tokenUrl` |
|---|---|---|
| `GitHub` | `https://github.com/login/oauth/authorize` | `https://github.com/login/oauth/access_token` |
| `Google` | `https://accounts.google.com/o/oauth2/v2/auth` | `https://oauth2.googleapis.com/token` |
| `Box` | `https://account.box.com/api/oauth2/authorize` | `https://api.box.com/oauth2/token` |
| `AzureActiveDirectory` / `aad3rdPartySNI` | `https://login.microsoftonline.com/common/oauth2/v2.0/authorize` | `https://login.microsoftonline.com/common/oauth2/v2.0/token` |

For `identityProvider` values not in this table (`dynamicscrmonlinecertificate`, `salesforce`, `dropbox`, `oauth2generic`, …), look the provider's well-known OAuth endpoints up in its developer docs — the catalog API does not surface them.

Use the `scopes` array from this response as the default scopes list. The catalog `clientId` is Microsoft's default OAuth App; replace it with your own only when going BYO.

Derive `authType` from `connectionParameters`:

- Any parameter with `type: oauthSetting` → `authType = OAuth2`.
- Else any parameter with `type: securestring` → `authType = CustomKeys`.
- Else → `authType = None` (anonymous) or `ProjectManagedIdentity` if the connector explicitly supports MI.

### 3. Logic Apps **apiOperations** — operation catalog (`gateway_connector` only)

For `gateway_connector` you need the list of operations the connector exposes plus each operation's parameter schema, because that's what gets serialized into `metadata.mcpserverConfigProperties` on PUT #2. Asset-gallery does not carry this.

```
GET https://management.azure.com/subscriptions/{sub}
    /providers/Microsoft.Web/locations/{region}/managedApis/{connectorName}
    /apiOperations?api-version=2016-06-01
```

Returns `value[]` of operations with `name`, `properties.summary` (display name), `properties.description`, `properties.annotation.family`, and `properties.visibility` (`important` / `advanced` / `internal`). Verified 2026-05-21: Box returns 14 operations including `ListRootFolder`, `ListFolder`, `GetFileMetadata`, `GetFileContent`, `DeleteFile`, `CreateFile`, plus several `On*` triggers (not agent-callable).

To get parameter schemas, fetch a single operation with `$expand=properties/inputsDefinition`:

```
GET .../managedApis/{connectorName}/apiOperations/{operationName}
    ?api-version=2016-06-01&$expand=properties/inputsDefinition
```

`properties.inputsDefinition` is a JSON-Schema-shaped object with `type:"object"`, `properties:{...}`, and `required:[...]`. Map each entry to one `agentParameters` entry:

| `inputsDefinition.properties[name]` field | → `agentParameters[].schema` field |
|---|---|
| `type` | `type` |
| `description` | `description` |
| `title` | `x-ms-summary` |
| `default` | `default` (omit if absent) |

If `inputsDefinition.properties` is empty / missing, the operation takes no arguments and `agentParameters` is `[]` (e.g. Box `ListRootFolder`).

Skip any operation whose `properties.isWebhook` or `isNotification` is `true` — these are Logic Apps triggers, not agent-callable actions.

**Picking ops from a plain-language scenario.** Match the user's words against `properties.summary` and `properties.description`, then prefer the simplest variant (fewest required parameters) and the one whose `annotation.family` aligns with the user intent. For Box "list my files", `ListRootFolder` (zero params) wins over `ListFolder` (requires `id`); if the user asks to list a specific folder, register both.

## Gateway connector full flow

For Catalog tiles tagged `Custom · Preview` (Box, Pipedrive, GitHub, Salesforce, Outlook, iManage Work, PDF4me, Qdrant, Medallia, Fulcrum, monday.com, SuperMCP, IA-Connect JML, iMIS, Huddo Boards, The Events Calendar, PUG Gamified Engagement, Nitro Sign Enterprise Verified, Soft1, Elfsquad Product Configurator, MintNFT, …).

### Step 1 — Discover

Query the asset-gallery (call #1) for the connector. Extract:

- `objectId` from `entityId` → `connectorName`
- Full `entityId` → `metadata.toolEntityId`

Then call managedApis (call #2) and apiOperations (call #3) for OAuth and operation metadata.

### Step 2 — PUT #1 (create connection)

Verbatim PUT body (captured from the portal's Box wizard, 2026-05-21):

```json
{
  "properties": {
    "authType": "OAuth2",
    "category": "RemoteTool",
    "target": "https://placeholder",
    "credentials": {},
    "connectorName": "box",
    "metadata": {
      "type": "gateway_connector",
      "toolEntityId": "azureml://location/eastus/apiCenter/connectors-registry-prod-bl/type/tools/objectId/box/version/1",
      "connectionproperties": "{\"connectorName\":\"box\"}"
    }
  }
}
```

Spelling traps (case-sensitive):

- `toolEntityId` — NOT `entityId`.
- `connectionproperties` — **lowercase**, value is a **stringified JSON object**, not a nested object. `"{\"connectorName\":\"box\"}"` is correct; `{"connectorName":"box"}` is rejected.
- `connectorName` appears at top-level under `properties` **and** inside `metadata` and inside `connectionproperties`.

`target = "https://placeholder"` is the **persisted value on PUT #1**, not a stub. There is no follow-up call that rewrites it before PUT #2. Runtime dispatch keys off `metadata.toolEntityId` + `metadata.connectionproperties.connectorName` + OAuth consent state. PUT #2 (register-actions) rewrites `target` to the real gateway URL `https://app-XX.<region>.logic.azure.com/api/connectorGateways/{envId}/mcpServerConfigs/{connectionName}/mcp`.

### Step 3 — Per-caller consent

For every distinct end-user (or service principal), call `listConsentLinks`:

```
POST .../connections/{name}/listConsentLinks?api-version=2025-04-01-preview
```

Verbatim portal body:

```json
{
  "parameters": [{
    "objectId":      "<caller AAD oid>",
    "parameterName": "token",
    "redirectUrl":   "https://ai.azure.com/nextgen/authConsentPopup",
    "tenantId":      "<caller AAD tid>"
  }]
}
```

Notes:

- The portal sends `redirectUrl=https://int.ai.azure.com/...` from the INT environment; for production (`ai.azure.com`) use `https://ai.azure.com/nextgen/authConsentPopup`. The redirect URL only gates which Foundry origin the OAuth popup closes back into — it does not affect what tokens are minted.
- Returns a per-user OAuth authorization URL (e.g. a `box.com/api/oauth2/authorize?...` link). User navigates → consents → gateway stores the token.
- Cross-tenant calls return `InvalidConsentLinkParameter` (`objectId` + `tenantId` must match the caller principal).

#### Consent link expiry (~1 hour)

Each `listConsentLinks` response mints a short-lived signed token (≈ 1 hour TTL based on `ExpirationTime` in the base64 payload). A `500` from the consent host when clicking the link is most often caused by an **expired or stale link**, not a server outage. Fix: call `listConsentLinks` again to get a fresh link and use it immediately. Do not reuse a link from a previous step or previous session.

#### Portal popup lifecycle (pending-true happy path)

The portal pre-opens a blank popup (`about:blank`) before calling `listConsentLinks`, then drives the flow as follows once the consent URL is in hand. Code-first callers should replicate this:

1. Register listeners on `window.postMessage` **and** `BroadcastChannel('connector-oauth-callback')` to receive completion signals.
2. Navigate the popup to the consent URL.
3. Poll `popup.closed` every 1 second to detect finish / dismiss.
4. When the popup closes, wait **500 ms** grace for any in-flight postMessage / BroadcastChannel messages.
5. If a `{ pending: true }` signal arrives (consent completed server-side but no authorization code returned to the opener):
   - Issue a **PUT** to the connection (same body as the original create PUT) to prompt the backend to finalise auth state.
   - If `overallStatus` is `Connected` in the response, done ✅.
   - Otherwise **poll `GET .../connections/{name}`** every 2 seconds, up to **15 attempts**, until `overallStatus` flips to `Connected`.
6. If **no signal** before popup close, treat as user-cancelled and surface an error.
7. **Cleanup:** remove listeners, clear polling, force-close the popup if still open.

The `{ pending: true }` path is the normal happy-path because the provider closes the popup by redirecting to `ai.azure.com/nextgen/authConsentPopup`, which has no JavaScript opener to post back to. **Don't assume consent is done just because the popup closed.** The "blank Foundry page" seen after authorising in a detached tab is this same redirect arriving without an opener — the gateway token is still stored; retry PUT #2 to confirm.

#### Consent-host hosts

Links served from `logic-apis-df.consent.azure-apim.net` are the **dogfood / INT** consent host (DF = dogfood). Production region traffic goes through `logic-apis-{region}.consent.azure-apim.net` (e.g. `logic-apis-eastus.consent.azure-apim.net`). Either host can return DF links depending on which Logic Apps environment the connector is deployed in; the caller cannot force the host.

#### Dogfood OAuth-app runtime allowlist trap

Some connectors (Spotify `spotifyip` confirmed) are backed by a **dogfood-env Microsoft OAuth app** registered in provider "development mode" with a hard-coded test-user allowlist. Consent + `Connected` status work fine code-first for any caller, but `tools/call` at runtime returns:

```json
{ "error": { "code": 403, "source": "...logic-df.azure-apihub.net",
  "innerError": "Check settings on https://developer.spotify.com/dashboard, the user may not be registered." } }
```

Detect by inspecting the consent URL's first 302: if the `redirect_uri` is `https://global-test.consent.azure-apim.net/redirect` (rather than `global.consent...`), the connector is on the dogfood OAuth app. **The connection will still go Connected and `tools/list` will work**; only the actual API invocation fails. Not fixable client-side; requires Microsoft to promote the app or add the caller's email to the provider-side allowlist.

```pwsh
$consentUrl = ($r.Content | ConvertFrom-Json).value[0].link
try { Invoke-WebRequest -Uri $consentUrl -MaximumRedirection 0 -ErrorAction Stop | Out-Null }
catch { $loc = $_.Exception.Response.Headers.Location.ToString() }
if ($loc -match 'global-test\.consent\.azure-apim\.net') {
  Write-Warning "Connector uses dogfood OAuth app; tools/call may 403 with 'user may not be registered' even after Connected."
}
```

### Step 4 — PUT #2 (register actions)

After OAuth, the portal issues a **second PUT** against the **same connection name** to register which connector operations the agent can invoke. **Without this PUT the runtime has no actions to dispatch even though `overallStatus` shows `Authenticated`.**

The body is identical to PUT #1 plus an additional `metadata.mcpserverConfigProperties` field (stringified JSON). Verbatim example for Box connection `box-5`:

```jsonc
{
  "properties": {
    "authType": "OAuth2",
    "category": "RemoteTool",
    "target": "https://placeholder",
    "credentials": {},
    "connectorName": "box",
    "metadata": {
      "type": "gateway_connector",
      "toolEntityId": "azureml://location/eastus/apiCenter/connectors-registry-prod-bl/type/tools/objectId/box/version/1",
      "connectionproperties": "{\"connectorName\":\"box\"}",
      "mcpserverConfigProperties": "{\"description\":\"\",\"state\":\"Enabled\",\"connectors\":[{\"name\":\"box\",\"connectionName\":\"box-5\",\"displayName\":\"box\",\"description\":\"\",\"operations\":[{\"name\":\"GetFileMetadata\",\"displayName\":\"Get file metadata using id\",\"description\":\"\",\"userParameters\":[],\"agentParameters\":[{\"name\":\"id\",\"schema\":{\"type\":\"string\",\"description\":\"The unique identifier of the file in Box.\",\"x-ms-summary\":\"File Id\"}}]}]}]}"
    }
  }
}
```

Decoded `mcpserverConfigProperties` schema:

```jsonc
{
  "description": "",
  "state": "Enabled",
  "connectors": [
    {
      "name":           "<connectorName>",       // same as properties.connectorName
      "connectionName": "<this connection name>",
      "displayName":    "<connectorName>",
      "description":    "",
      "operations": [
        {
          "name":            "<OperationId>",    // operation id from apiOperations
          "displayName":     "<friendly>",
          "description":     "",
          "userParameters":  [],                   // bound at connection time (rare for Custom·Preview)
          "agentParameters": [                     // parameters the agent fills at call time
            {
              "name": "<paramName>",
              "schema": {
                "type":         "string|number|boolean",
                "description":  "...",
                "x-ms-summary": "...",
                "default":      "..."              // optional
              }
            }
          ]
        }
      ]
    }
  ]
}
```

Each operation in `operations[]` corresponds 1:1 to one `apiOperations` entry; `agentParameters[].schema` is translated from `inputsDefinition.properties` per the mapping in [Catalog APIs §3](#3-logic-apps-apioperations--operation-catalog-gateway_connector-only).

The portal lets the user multi-select via checkboxes in the wizard's "Configure actions" page; the selection is serialized into this string. When the selection changes later, the portal **replaces `mcpserverConfigProperties` wholesale** — no merge. Your code must do the same: any time the agent-callable op list changes, re-run PUT #2 with the full new list.

### Step 5 — `overallStatus` flip semantics

Two independent conditions must BOTH be true for `overallStatus` to flip `Unauthenticated` → `Connected`:

1. **PUT #2 issued with non-empty `metadata.mcpserverConfigProperties`** (rewrites `target` to the real gateway URL; target rewrite is visible immediately on PUT #2 regardless of consent state).
2. **OAuth consent completed** (user followed the `listConsentLinks` URL and clicked Authorize). Gateway then stores the token.

Order-independent observations:

- PUT #2 before consent → `target` rewrites, status stays `Unauthenticated`.
- Consent before PUT #2 → status stays `Unauthenticated` until PUT #2 fires; PUT #2 then flips to `Connected` in the same response.

## Body shape — `OAuth2` + `catalog_MCP` (Microsoft-managed OAuth)

Use when the catalog entry is an MCP server and you accept Microsoft's managed OAuth App + consent flow (no BYO secret):

```json
{
  "properties": {
    "authType": "OAuth2",
    "category": "RemoteTool",
    "target": "https://api.githubcopilot.com/mcp",
    "credentials": {},
    "metadata": {
      "type": "catalog_MCP",
      "toolEntityId": "azureml://location/eastus/apiCenter/connectors-registry-prod-bl/type/tools/objectId/github/version/1"
    },
    "peRequirement": "NotRequired"
  }
}
```

For MCP URL discovery when `connectors-registry-prod-bl` lacks `remotes[]`, look up the peer entry in `registry-prod-bl` (e.g. `github-mcp-server`) — its asset-gallery row sometimes carries the canonical MCP URL. Consent uses the same `listConsentLinks` flow as gateway_connector.

## BYO OAuth App against a catalog MCP server

When the user has their own OAuth App (e.g. GitHub `https://github.com/organizations/<org>/settings/applications/<app-id>`) and wants the connection to mint tokens via *their* app instead of Microsoft's managed one. Verified shape, 2026-05-21:

```json
{
  "properties": {
    "authType": "OAuth2",
    "category": "RemoteTool",
    "target": "<MCP server URL>",
    "credentials": { "clientId": "<your client id>", "clientSecret": "<your client secret>" },
    "authorizationUrl": "https://github.com/login/oauth/authorize",
    "tokenUrl":         "https://github.com/login/oauth/access_token",
    "scopes":           ["repo","workflow","read:org","admin:org"],
    "peRequirement":    "NotRequired"
  }
}
```

### Filling the OAuth fields from the catalog APIs

1. **Find the connector `entityId`** — asset-gallery POST with `annotations/name contains <name>`. Pull `objectId` out of the returned `entityId`.
2. **Look up OAuth metadata** — `GET .../managedApis/<objectId>?api-version=2016-06-01`. From `properties.connectionParameters.token.oAuthSettings`:
   - `identityProvider` → look up `authorizationUrl` / `tokenUrl` in the mapping table in [Catalog APIs §2](#2-logic-apps-managedapis--oauth-source-of-truth).
   - `scopes` → use as the default scopes array (override only if the user explicitly needs different scopes).
3. **Supply your own `clientId` / `clientSecret`** in `credentials`. Do not reuse the catalog `clientId` from step 2 — that's Microsoft's managed OAuth App and you cannot mint with it.
4. **PUT** the body above.

### Hard rules verified by probe (2026-05-21)

- **`scopes` MUST be a JSON array.** A space-separated string returns `400 "Error when parsing request; unable to deserialize request body"`.
- **DO NOT send `useCustomConnector`.** It is ignored on input; server fills `false`.
- **DO NOT send `metadata.{type=catalog_MCP, toolEntityId, ...}`** for BYO. Those fields anchor the connection to the catalog's managed OAuth App and conflict with your supplied `credentials`.
- **DO NOT call `listConsentLinks`** for BYO — the gateway handles consent via the standard authorization_code flow using the server-filled `redirectUrl`. Calling `listConsentLinks` against a fresh BYO connection returns `404 AIGatewayConnectionNotFound`.

### Server-filled response fields

- `credentials` → `null` (scrubbed).
- `connectorName` → `<gatewayId>-<connectionName>` (your input ignored).
- `redirectUrl` → `https://global.consent.azure-apim.net/redirect/<32-hex>` — **the OAuth callback URL the provider (e.g. GitHub OAuth App) must allow-list**. Generated per-connection on first PUT. Two-pass flow:
  1. PUT with placeholder client_secret.
  2. Read `properties.redirectUrl` from the response.
  3. Register it as the "Authorization callback URL" on the OAuth App.
  4. PUT again with the real client_secret.

### Caveat: `api.githubcopilot.com/mcp` rejects BYO OAuth-App tokens

The GitHub Copilot MCP server requires GitHub-App-minted Copilot tokens (the `microsoft-foundry-agent-service` GitHub App). A token from a user OAuth App will be rejected at runtime even if the connection PUT is 200. For real BYO testing point `target` at a self-hosted GitHub MCP server, or use an OpenAPI tool against `api.github.com` instead.

## `ProjectManagedIdentity` Remote MCP

For MCP servers that accept Azure-side identity (the project's system MI calls the MCP server's bearer endpoint):

```json
{
  "properties": {
    "authType": "ProjectManagedIdentity",
    "category": "RemoteTool",
    "target": "<MCP server URL with required query string>",
    "metadata": { "type": "generic_mcp", "audience": "<upstream resource URI>" }
  }
}
```

For catalog-listed MCP servers, prefer `metadata.type = catalog_MCP` with `toolEntityId` so `target` is prepopulated. `audience` is **required for MI auth** — it tells Foundry which resource URI to request a token for. Read the required `audience` from the connector's catalog entry or its documentation (typical values: an app ID URI like `api://contoso-mcp`, or an Azure service resource ID like `https://cognitiveservices.azure.com`). If you omit `audience`, the MCP server rejects the call with 401.

### `ProjectManagedIdentity` limitations

Verified end-to-end against Azure Language `/language/mcp`, 2026-05-21:

1. **Forwarder drops the query string.** The connection `target`'s `?api-version=...` is **not** preserved on the upstream call. If the upstream MCP requires a query parameter, PMI fails with 401/404 even when RBAC is correct.
2. **Forwarder mints the wrong audience.** The MI token Foundry sends does not have `aud=https://cognitiveservices.azure.com` or `https://ai.azure.com`. Setting `properties.audience` on the connection is accepted but **does not** change what is minted.
3. Endpoints not on the trust list reject the forwarded MI token with `-32007 PERMISSION_DENIED "Cannot pass Microsoft token to untrusted MCP endpoint"` (e.g. `api.githubcopilot.com/mcp`). This is the expected security gate.

## `CustomKeys` Remote MCP

Static header(s) injected on every upstream call. Minimum body:

```json
{
  "properties": {
    "authType": "CustomKeys",
    "category": "RemoteTool",
    "target": "<MCP server URL>",
    "credentials": { "keys": { "Ocp-Apim-Subscription-Key": "<value>" } },
    "metadata": { "type": "generic_mcp" }
  }
}
```

Verified PUT 200 / GET 200 / DELETE 200 round-trip. The header name is **arbitrary** — it is forwarded as-is to the MCP server. Different connectors require different header shapes:

- GitHub PAT: `Authorization: Bearer <pat>` or `Authorization: token <pat>` — catalog dictates.
- API-key services: `x-api-key: <key>` or `Ocp-Apim-Subscription-Key: <key>`.
- Multi-header schemes: e.g. `X-Account-Id: <id>` + `X-Account-Secret: <secret>`.

Always read the canonical header set from the connector's `connectionParameters` (each `securestring` parameter names the header it maps to) before writing the `keys` block. **Do not default to `Authorization: Bearer`** — it's wrong for many connectors.

For catalog-listed servers, swap `metadata.type` to `catalog_MCP` and add `toolEntityId`.

## `UserEntraToken` Remote MCP

For MCP servers that consume the *caller's* Entra token directly. Body includes `metadata.audience` so the platform mints the correct token for the upstream:

```json
{
  "properties": {
    "authType": "UserEntraToken",
    "category": "RemoteTool",
    "target": "<MCP server URL>",
    "metadata": { "type": "generic_mcp", "audience": "<upstream resource URI>" }
  }
}
```

Not available when the agent is published to Teams (Teams agents use the project MI).

## Custom subtab — OpenAPI / A2A

Not catalog-driven — the user provides the spec themselves. Each Save in this subtab maps to a single PUT against the same connections endpoint:

| Tile | `authType` options | `target` | Notes |
|---|---|---|---|
| OpenAPI | `None`, `CustomKeys`, `ApiKey`, `OAuth2` | OpenAPI spec URL or upstream API base | Agent gets `tools[].openapi.auth.security_scheme.connection_id`. |
| A2A (Preview) | `None` / `CustomKeys` / `UserEntraToken` / `AAD` (mapped from UI) | A2A endpoint | `metadata.agentCardPath` default `/.well-known/agent-card.json`; agent gets `tools=[A2APreviewTool(project_connection_id=...)]`; runtime emits `a2a_preview_call` / `a2a_preview_call_output` events. |
| MCP | covered above | — | This tile is just a router to the catalog / BYO flows. |

## Toolbox attach — `gateway_connector` tool naming

Attach the same as `generic_mcp` — the tool block uses `type:"mcp"` and `project_connection_id` set to the **full ARM resource id** of the connection (NOT just the name):

```jsonc
{
  "tools": [{
    "type": "mcp",
    "server_label": "box5",
    "project_connection_id":
      "/subscriptions/{sub}/resourceGroups/{rg}/providers/Microsoft.CognitiveServices/accounts/{acct}/projects/{proj}/connections/{connName}"
  }]
}
```

`tools/list` returns one MCP tool per registered operation. Tool names follow the verified pattern (probed 2026-05-22 against Box):

```
<server_label>___<connectorName>_<OperationName>
```

Note `___` (**three** underscores) between `server_label` and the rest, then a **single** `_` between `connectorName` and operation name. Example for Box attached with `server_label="box5"`:

| `mcpserverConfigProperties` op | `tools/list` `name` | `description` |
|---|---|---|
| `ListRootFolder` | `box5___box_ListRootFolder` | `box - List files and folders in root folder` |
| `GetFileMetadata` | `box5___box_GetFileMetadata` | `box - Get file metadata using id` |

The MCP tool's `inputSchema` is exactly the JSON schema derived from `apiOperations/{op}?$expand=properties/inputsDefinition` (the `agentParameters[].schema` values, re-keyed by parameter name). For an operation with no agent parameters, `inputSchema` is `{"type":"object"}`.

Worked `tools/call` for "list my files in Box" — verified end-to-end:

```jsonc
POST {dp}/toolboxes/{tb}/mcp?api-version=v1
{
  "jsonrpc": "2.0", "id": 2, "method": "tools/call",
  "params": { "name": "box5___box_ListRootFolder", "arguments": {} }
}
→ 200
{
  "jsonrpc": "2.0", "id": 2,
  "result": {
    "content": [{ "type": "text", "text": "[]" }],
    "isError": false
  }
}
```

(`text` carries a JSON-stringified array of Box file/folder objects; empty `[]` means the root folder is empty.)

### `outlook` connector — verified end-to-end (2026-05-22)

Uses `identityProvider: oauth2generic` (MSA / consumers tenant). `connectorName = "outlook"`, `toolEntityId` objectId = `outlook`. `tools/call` response wraps in `{ "value": [...] }` (not a bare array like Box):

```jsonc
POST {dp}/toolboxes/{tb}/mcp?api-version=v1
{
  "jsonrpc": "2.0", "id": 2, "method": "tools/call",
  "params": { "name": "outlook-1___outlook_GetEmailsV2",
              "arguments": { "folderPath": "Inbox", "top": 3 } }
}
→ 200
{
  "jsonrpc": "2.0", "id": 2,
  "result": {
    "content": [{ "type": "text",
      "text": "{\n  \"value\": [\n    { \"Subject\": \"...\", \"From\": \"...\", ... }\n  ]\n}" }],
    "isError": false
  }
}
```

Operations registered for the test: `GetEmailsV2` (read emails with `folderPath` / `top` / `fetchOnlyUnread` agent parameters) and `SendEmailV2` (send with `emailMessage` object param containing required `To`, `Subject`, `Body`). `SendEmailV2`'s top-level schema is `object` — pass it as a single nested `agentParameters` entry; the gateway flattens into the Logic Apps `emailMessage` envelope internally. The follow-up PUT after popup close (pending-true path) immediately returned `overallStatus: Connected` without needing the GET poll loop — outlook's MSA consent round-trips are fast.

## Minimum attach + verify recipe

Verifying a fresh connection is the only toolbox operation in scope of this reference. Toolboxes are upserted implicitly by `POST /versions`; no separate container create is needed.

The `$dp` value below is the project's data-plane endpoint, in the same `{project_endpoint}` form used elsewhere in these references — `https://<account>.services.ai.azure.com/api/projects/<project>`. The host segment varies by Foundry account/region; read it from a non-`FOUNDRY_`-prefixed env var (see [toolbox-reference.md § Agent env contract](toolbox-reference.md#agent-env-contract)) rather than hardcoding. The bearer-token resource is `https://ai.azure.com`, NOT ARM.

```pwsh
# 0. Constants.
$dp   = $env:PROJECT_ENDPOINT   # https://<account>.services.ai.azure.com/api/projects/<project>
$tb   = "default-tb"
$lbl  = "box5"                  # becomes the "<label>___" prefix on tool names
$connId = "/subscriptions/<sub>/resourceGroups/<rg>/providers/Microsoft.CognitiveServices/accounts/<acct>/projects/<proj>/connections/<connName>"
$tok  = az account get-access-token --resource "https://ai.azure.com" --query accessToken -o tsv
$hdr  = @{ Authorization      = "Bearer $tok"
           "Content-Type"     = "application/json"
           Accept             = "application/json, text/event-stream" }

# 1. Create a toolbox version with the connection attached.
$body = @{ tools = @(@{
   type = "mcp"
   server_label = $lbl
   project_connection_id = $connId
}) } | ConvertTo-Json -Depth 6 -Compress
$v = Invoke-WebRequest -Method POST -Headers $hdr -UseBasicParsing -Body $body `
        -Uri "$dp/toolboxes/$tb/versions?api-version=v1"
$ver = ($v.Content | ConvertFrom-Json).version

# 2. Promote the new version to default. default_version MUST be a JSON STRING, not a number.
#    Use ${tb} to terminate the variable name unambiguously before the literal '?'.
Invoke-WebRequest -Method PATCH -Headers $hdr -UseBasicParsing `
   -Body (@{ default_version = "$ver" } | ConvertTo-Json) `
   -Uri "$dp/toolboxes/${tb}?api-version=v1" | Out-Null

# 3. tools/list → expect one entry per registered op, named "<server_label>___<connectorName>_<OpName>".
$req = '{"jsonrpc":"2.0","id":1,"method":"tools/list","params":{}}'
Invoke-WebRequest -Method POST -Headers $hdr -UseBasicParsing -Body $req `
   -Uri "$dp/toolboxes/$tb/mcp?api-version=v1"

# 4. tools/call → the prefixed name and arguments per inputSchema.
$call = @{ jsonrpc="2.0"; id=2; method="tools/call"; params=@{
   name="$lbl`___box_ListRootFolder"; arguments=@{} } } | ConvertTo-Json -Depth 5 -Compress
Invoke-WebRequest -Method POST -Headers $hdr -UseBasicParsing -Body $call `
   -Uri "$dp/toolboxes/$tb/mcp?api-version=v1"
```

The response body for `/mcp` is plain JSON (no SSE `data:` framing) despite the `text/event-stream` Accept.

## Required RBAC summary

| Operation | Role |
|---|---|
| PUT any connection above | **Azure AI Developer** on the project (or **Cognitive Services Contributor** on the account) |
| Drive OAuth consent (`gateway_connector`, `catalog_MCP` managed-OAuth) | The end-user themselves, signed in to the subscription's tenant |
| `ProjectManagedIdentity` against a Cognitive Services upstream | Project MI needs the upstream's data-plane role (e.g. `Cognitive Services Language Owner` for `/language/mcp`) |

## Pitfalls / common mistakes

- **Do not forget PUT #2 for `gateway_connector`** ([Step 4](#step-4--put-2-register-actions)). The first PUT + OAuth flips status to `Authenticated` but the runtime has no actions to dispatch until you PUT again with `metadata.mcpserverConfigProperties`.
- **Do not invent a "real" target URL** for the `gateway_connector` flow on PUT #1. `"https://placeholder"` is correct on PUT #1; PUT #2 rewrites it.
- **Do not mix BYO `credentials` with `metadata.type=catalog_MCP`** in the BYO body. They conflict; the server accepts the PUT but the runtime uses the catalog's managed app and ignores your secret — or fails with consent confusion.
- **Do not send `scopes` as a space-separated string** anywhere. Always an array.
- **Do not call `listConsentLinks` for BYO OAuth.** Use only for `gateway_connector` and managed-OAuth `catalog_MCP`.
- **Do not assume the asset-gallery search response contains OAuth metadata** — it does not. Always pair it with the Logic Apps `managedApis` GET (or hardcode the identityProvider mapping) to get `scopes` and to derive `authorizationUrl` / `tokenUrl`.
- **Use exact field spelling** for `gateway_connector`: `toolEntityId` (NOT `entityId`), `connectionproperties` (lowercase, stringified JSON).
- **Sign in to the subscription's tenant** before calling `listConsentLinks` — it validates the caller principal owns the supplied `objectId` + `tenantId`.
- **Toolbox PATCH `default_version` must be a JSON STRING**, not a number. Sending `{"default_version": 1}` returns `400 invalid_payload "requires an element of type 'String', but the target element has type 'Number'"`. Use `{"default_version": "1"}`.
- **`metadata.audience` is required for `ProjectManagedIdentity`.** Without it the MCP server returns 401.
- **Header names for `CustomKeys` come from the catalog**, not from a default `Authorization: Bearer` template.
- **`ApiKey` is rejected** for `category=RemoteTool`. Use `CustomKeys` for static secrets.
- **OAuth consent is per-user, per-connection, per-project.** Each new caller hits `CONSENT_REQUIRED` once (returned as a nested string code inside an outer `-32006` error) and must open the URL the toolbox returns.
- **`api.githubcopilot.com/mcp` rejects user OAuth-App tokens.** Use a self-hosted MCP or fall back to OpenAPI.
- **PMI forwarder drops `target` query strings and mints a fixed audience.** Setting `properties.audience` is accepted but does not change what is sent.
- **Network-secured Foundry** projects cannot use private-endpoint-only MCP servers — only public endpoints reachable from the Foundry data plane and the Connector Namespace.

## References

- [Tool Catalog](https://learn.microsoft.com/azure/foundry/agents/concepts/tool-catalog)
- [Toolbox (preview)](https://learn.microsoft.com/azure/foundry/agents/how-to/tools/toolbox)
- [Private tools catalog](https://learn.microsoft.com/azure/foundry/agents/concepts/tool-catalog#private-tools-catalog)
- [Cognitive Services projects REST API](https://learn.microsoft.com/rest/api/aiservices/)
- [tool-mcp.md](tool-mcp.md) — prompt-agent MCP wiring (no toolbox)
- [toolbox-reference.md](toolbox-reference.md) — MCP endpoint, auth, testing, troubleshooting
- [agent-tools.md](agent-tools.md) — the agent-tools index
- [use-toolbox-in-hosted-agent.md](use-toolbox-in-hosted-agent.md) — wiring a toolbox into a hosted agent

# Local Run Reference

Use this when iterating on a hosted agent before deploying.

> **Prerequisite:** Local run does NOT require `azd provision` or any deployed Azure infrastructure. The agent runs on your machine and calls the Foundry model endpoint directly using your local credentials (`DefaultAzureCredential` — falls back to `az login` / VS Code identity). You only need a `.env` file in the agent directory with:
> ```env
> FOUNDRY_PROJECT_ENDPOINT=https://<account>.services.ai.azure.com/api/projects/<project>
> AZURE_AI_MODEL_DEPLOYMENT_NAME=<model-deployment-name>
> ```
> If you already ran `azd provision`, extract these from `azd env get-values`.
>
> 🚦 **If no project endpoint is configured (not in the message, `azd env`, or `.env`) and the user hasn't asked to create one, stop and ask them to pick an existing project or confirm creating a new one — don't silently select or `azd provision` one.** Once they choose, follow [deploy.md Step 2](../../deploy/deploy.md#step-2----provision-azure-resources-one-time-per-env) to provision or resolve the project, then return here for local iteration before deploying the agent.
>
> **Critical: keep `.env` and `azd env` in sync.** `azd ai agent run` injects the active `azd env` values into the agent process before Python loads `.env`. Many samples use `load_dotenv(override=False)`, so an existing process environment value wins over `.env`. If you change the project endpoint or model deployment, update both `.env` and `azd env`:
> ```bash
> azd env set FOUNDRY_PROJECT_ENDPOINT "https://<account>.services.ai.azure.com/api/projects/<project>"
> azd env set AZURE_AI_MODEL_DEPLOYMENT_NAME "<model-deployment-name>"
> azd env get-values
> ```
> A stale `AZURE_AI_MODEL_DEPLOYMENT_NAME` in `azd env` can make local run call the wrong deployment even when `.env` is correct, commonly surfacing as a Foundry responses API `404 Not Found`.

## Prepare the local environment

For Python agents, prepare the environment from the **agent's service source directory** -- the folder that contains `requirements.txt` and the agent source (typically `<repo>/src/<service-name>/`, not the azd project root). `azd ai agent run` resolves the venv relative to this folder; a `.venv` created in the project root is ignored and azd silently creates a second one without `uv`.

1. `cd` into the service source directory.
2. Create a venv, for example `python -m venv .venv`.
3. Activate the venv.
4. Install `uv` inside the active venv: `python -m pip install uv`.
5. In the same shell with the service-dir `.venv` activated, run `azd ai agent run` (from any cwd in the project); it installs `requirements.txt` itself and uses `uv` from the active venv for faster Python dependency installation.

> **Important:** The venv must live next to `requirements.txt`, not in the azd project root. Install `uv` before running `azd ai agent run`, and keep that venv activated when running the command; otherwise the local run falls back to slower dependency installation. Do NOT manually run `pip install -r requirements.txt` / `uv pip install -r requirements.txt --prerelease=allow`; let `azd ai agent run` install dependencies.

## Start the agent locally

Activate the service-dir `.venv`, then in that venv run:

```bash
azd ai agent run
```

What this does:

1. Resolves the agent service from `azure.yaml` (auto-picks when only one exists).
2. Detects the project type (Python, .NET) from files in the service source dir.
3. Installs dependencies if needed. For Python, `azd ai agent run` installs `requirements.txt` itself and uses `uv` from the active local environment when available.
4. Starts the agent in the foreground on `localhost:8088` (default).
5. Opens **Agent Inspector** in your browser (unless `--no-inspector`).

> Wait for the ready log line before sending the first invocation. Poll the log at short intervals; do not pre-sleep on a fixed duration.

`Ctrl+C` stops the agent and clears the saved local session id in an interactive terminal.

For headless or CI runs, pass `--no-inspector` and start the local server in a managed background session that later steps can monitor and stop. Wait for the ready log line, invoke it from a second command, then stop the same background session before deploying or leaving a temporary workspace.

Do **not** start `azd ai agent run` as a detached process that you cannot monitor or stop (for example, a bare `azd ai agent run ... &`, or a popped PowerShell window on Windows). Keep logs, readiness polling, and the PID/process handle for cleanup.

## Useful flags

| Flag | Purpose |
|------|---------|
| `--port <n>` / `-p <n>` | Override the listen port. Useful when 8088 is taken. |
| `--start-command "<cmd>"` / `-c "<cmd>"` | Override `azure.yaml` and auto-detect. Example: `--start-command "python app.py"`. |
| `--no-inspector` | Skip opening Agent Inspector. Use in CI / SSH. |

Pass the service name when there are multiple `ai.agent` services:

```bash
azd ai agent run my-agent
```

## Where the start command comes from

Resolution order (first non-empty wins):

1. `--start-command` flag.
2. `azure.yaml services.<name>.config.startupCommand`.
3. Auto-detected from project type.

Example:

```yaml
# azure.yaml
services:
  my-agent:
    project: src/my-agent
    language: python
    host: azure.ai.agent
    config:
      startupCommand: "uvicorn app:app --host 0.0.0.0 --port 4001"
```

If detection fails and no override is set, `run` errors with the project dir and asks for `--start-command` or `startupCommand`.

## Invoke the local agent

```bash
azd ai agent invoke --local "hello, are you up?"
```

Do not use `--output json` with invoke. The invoke command supports `default` and `raw` output only.

If the user did not explicitly specify a prompt, use `"hello, are you up"` for the local smoke test; only verify that the agent can return a response.

Run one representative local invocation before deploying. If the local invocation returns a model `404` or wrong deployment error, check `azd env get-values` before changing code; stale azd env values are the most common cause.

`--local` differs from a remote invoke in:

- Targets `http://localhost:<port>` instead of the Foundry endpoint.
- Skips the confirmation envelope (no billing, no remote mutation).
- `--version` is rejected (versions are a remote concept).
- Named-agent invocation is rejected (only one agent runs locally at a time).

Other useful flags:

| Flag | Purpose |
|------|---------|
| `--protocol responses` (default) / `--protocol invocations` | Wire format your agent speaks. |
| `--input-file request.json` / `-f request.json` | Send a file body instead of a string message. |
| `--new-session` | Drop the saved local session and start fresh. |
| `--port <n>` | Match the port you started `run` with. |

After the local invocation completes, stop the `azd ai agent run` process you started before moving on.

## When to graduate to remote

Local dev validates code shape; remote validates infra + identity + Foundry binding. Move to deploy when:

- You changed the agent's `model`, `tools`, `connections`, or `protocols` in `azure.yaml`. Those only take effect on the deployed agent.
- You need to test against real Foundry connections (search indexes, Bing, MCP, A2A) that have no local mock.
- You are ready to publish a new immutable agent version.

Before proceeding to deploy, clean up the local agent process.

Next step -> [deploy/deploy.md](../../deploy/deploy.md).

## Common failures

| Symptom | Likely cause | Fix |
|---------|--------------|-----|
| `could not connect to localhost:<port>` | `run` not started, or wrong port | Start `azd ai agent run`; pass `--port` to `invoke --local` if non-default. |
| `could not detect project type in <dir>` | Missing project marker file | Set `startupCommand` in `azure.yaml` or pass `--start-command`. |
| `cannot use --local with a named agent` | Named-agent invoke against localhost | Drop the name; only one local agent at a time. |
| `cannot use --version with --local` | `--version` is remote-only | Drop `--version`, or remove `--local` to hit the deployed agent. |
| Inspector never opens | Headless env, or extension install failed | Pass `--no-inspector`, or run `azd extension install azure.ai.inspector`. |
| Auth / connection errors against Azure services | Local credentials not wired | Expected -- `DefaultAzureCredential` falls back to your `az login` / VS Code identity. Use `azd auth login` if needed. |

# SDK Operations for Foundry Agent Service

Use the Foundry MCP tools for agent CRUD operations. When MCP tools are unavailable, use the `azure-ai-projects` Python SDK or REST API.

## Agent Operations via MCP

| Operation | MCP Tool | Description |
|-----------|----------|-------------|
| Create/Update agent | `agent_update` | Create a new agent or update an existing one (creates new version) |
| List/Get agents | `agent_get` | List all agents, or get a specific agent by name |
| Delete agent | `agent_delete` | Delete an agent |
| Invoke agent | `agent_invoke` | Send a message to an agent and get a response |
| Get schema | `agent_definition_schema_get` | Get the full JSON schema for agent definitions |

## SDK Agent Operations

When MCP tools are unavailable, use the `azure-ai-projects` Python SDK (`pip install azure-ai-projects --pre`):

```python
from azure.ai.projects import AIProjectClient
from azure.identity import DefaultAzureCredential

endpoint = "https://<resource>.services.ai.azure.com/api/projects/<project>"
client = AIProjectClient(endpoint=endpoint, credential=DefaultAzureCredential())
```

| Operation | SDK Method |
|-----------|------------|
| Create | `client.agents.create_version(agent_name, definition)` |
| List | `client.agents.list()` |
| Get | `client.agents.get(agent_name)` |
| Update | `client.agents.create_version(agent_name, definition)` (creates new version) |
| Delete | `client.agents.delete(agent_name)` |
| Chat | `client.get_openai_client().responses.create(model=<deployment>, input=<text>, extra_body={"agent": {"name": agent_name, "type": "agent_reference"}})` |

## Environment Variables

| Variable | Description |
|----------|-------------|
| `PROJECT_ENDPOINT` | Foundry project endpoint (`https://<resource>.services.ai.azure.com/api/projects/<project>`) |
| `MODEL_DEPLOYMENT_NAME` | Deployed model name (e.g., `gpt-4.1-mini`) |

## References

- [Agent quickstart](https://learn.microsoft.com/azure/ai-foundry/agents/quickstart?view=foundry)
- [Create agents](https://learn.microsoft.com/azure/ai-foundry/agents/how-to/create-agent?view=foundry)
- [Tool Catalog](https://learn.microsoft.com/azure/ai-foundry/agents/concepts/tool-catalog?view=foundry)

# Tool — Agent-to-Agent (A2A, preview)

Call another Foundry agent as if it were a tool. Useful for composing specialist agents into an orchestrator.

## Toolbox shape

```json
{
  "type": "a2a_preview",
  "name": "<AGENT_NAME>",
  "description": "<what this agent does>",
  "base_url": "<AGENT_BASE_URL>",
  "project_connection_id": "<connection_to_target_project>"
}
```

Auth is either anonymous (for the same project) or via a project connection that holds credentials for the remote agent's host.

## References

- [A2A tool documentation](https://learn.microsoft.com/azure/foundry/agents/how-to/tools/agent-to-agent)
- [agent-tools.md](agent-tools.md) — tool index

# Azure AI Search Tool

Ground agent responses with data from an Azure AI Search vector index. Requires a project connection and proper RBAC setup.

## Prerequisites

- Azure AI Search index with vector search configured:
  - One or more `Edm.String` fields (searchable + retrievable)
  - One or more `Collection(Edm.Single)` vector fields (searchable)
  - At least one retrievable text field with content for citations
  - A retrievable field with source URL for citation links
- A [project connection](../../../project/connections.md) between your Foundry project and search service
- `azure-ai-projects` package (`pip install azure-ai-projects --pre`)

## Required RBAC Roles

For **keyless authentication** (recommended), assign these roles to the **Foundry project's managed identity** on the Azure AI Search resource:

| Role | Scope | Purpose |
|------|-------|---------|
| **Search Index Data Contributor** | AI Search resource | Read/write index data |
| **Search Service Contributor** | AI Search resource | Manage search service config |

> **If RBAC assignment fails:** Ask the user to manually assign roles in Azure portal → AI Search resource → Access control (IAM). They need Owner or User Access Administrator on the search resource.

## Connection Setup

A project connection between your Foundry project and the Azure AI Search resource is required. See [Project Connections](../../../project/connections.md) for connection management via Foundry MCP tools.

## Query Types

| Value | Description |
|-------|-------------|
| `SIMPLE` | Keyword search |
| `VECTOR` | Vector similarity only |
| `SEMANTIC` | Semantic ranking |
| `VECTOR_SIMPLE_HYBRID` | Vector + keyword |
| `VECTOR_SEMANTIC_HYBRID` | Vector + keyword + semantic (default, recommended) |

## Tool Parameters

| Parameter | Required | Description |
|-----------|----------|-------------|
| `project_connection_id` | Yes | Connection ID (resolve via `project_connection_get`, typically after discovering the connection with `project_connection_list`) |
| `index_name` | Yes | Search index name |
| `top_k` | No | Number of results (default: 5) |
| `query_type` | No | Search type (default: `vector_semantic_hybrid`) |
| `filter` | No | OData filter applied to all queries |

## Limitations

- Only **one index per tool** instance. For multiple indexes, use connected agents each with their own index.
- Search resource and Foundry agent must be in the **same tenant**.
- Private AI Search resources require **standard agent deployment** with vNET injection.

## Troubleshooting

| Error | Cause | Fix |
|-------|-------|-----|
| 401/403 accessing index | Missing RBAC roles | Assign `Search Index Data Contributor` + `Search Service Contributor` to project managed identity |
| Index not found | Name mismatch | Verify `AI_SEARCH_INDEX_NAME` matches exactly (case-sensitive) |
| No citations in response | Instructions don't request them | Add citation instructions to agent prompt |
| Wrong connection endpoint | Connection points to different search resource | Re-create connection with correct endpoint |

## References

- [Azure AI Search tool documentation](https://learn.microsoft.com/azure/ai-foundry/agents/how-to/tools/azure-ai-search?view=foundry)
- [Tool Catalog](https://learn.microsoft.com/azure/ai-foundry/agents/concepts/tool-catalog?view=foundry)
- [Project Connections](../../../project/connections.md)

# Bing Grounding Tool

Access real-time web information via Bing Search. Unlike the [Web Search tool](tool-web-search.md) (which works out of the box), Bing Grounding requires a dedicated Bing resource and a project connection.

> ⚠️ **Warning:** Use the [Web Search tool](tool-web-search.md) as the default for web search. Only use Bing Grounding when the user **explicitly** requests Grounding with Bing Search or Grounding with Bing Custom Search.

## When to Use

- User explicitly asks for "Bing Grounding" or "Grounding with Bing Search"
- User explicitly asks for "Bing Custom Search" or "Grounding with Bing Custom Search"
- User needs to restrict web search to specific domains (Bing Custom Search)
- User has an existing Bing Grounding resource they want to use

## Prerequisites

- A [Grounding with Bing Search resource](https://portal.azure.com/#create/Microsoft.BingGroundingSearch) in Azure portal
- `Contributor` or `Owner` role at subscription/RG level to create Bing resource and get keys
- `Foundry Project Manager` role on the project to create a connection
- A project connection configured with the Bing resource key — see [connections](../../../project/connections.md)

## Setup

1. Register the Bing provider: `az provider register --namespace 'Microsoft.Bing'`
2. Create a Grounding with Bing Search resource in the Azure portal
3. Create a project connection with the Bing resource key — see [connections](../../../project/connections.md)
4. Set `BING_PROJECT_CONNECTION_NAME` environment variable

## Important Disclosures

- Bing data flows **outside Azure compliance boundary**
- Review [Grounding with Bing terms of use](https://www.microsoft.com/bing/apis/grounding-legal-enterprise)
- Not supported with VPN/Private Endpoints
- Usage incurs costs — see [pricing](https://www.microsoft.com/bing/apis/grounding-pricing)

## Troubleshooting

| Issue | Cause | Resolution |
|-------|-------|------------|
| Connection not found | Name mismatch or wrong project | Use `project_connection_list` to find the correct `connectionName` |
| Unauthorized creating connection | Missing Foundry Project Manager role | Assign role on the Foundry project |
| Bing resource creation fails | Provider not registered | Run `az provider register --namespace 'Microsoft.Bing'` |
| No results returned | Connection misconfigured | Verify Bing resource key and connection setup |

## References

- [Bing Grounding tool documentation](https://learn.microsoft.com/azure/ai-foundry/agents/how-to/tools/bing-grounding?view=foundry)
- [Tool Catalog](https://learn.microsoft.com/azure/ai-foundry/agents/concepts/tool-catalog?view=foundry)
- [Grounding with Bing Terms](https://www.microsoft.com/bing/apis/grounding-legal-enterprise)
- [Connections Guide](../../../project/connections.md)
- [Web Search Tool (default)](tool-web-search.md)

# Tool — Code Interpreter

Enables agents to write and run Python in a sandboxed environment. Supports data analysis, chart generation, and file processing. Has [additional charges](https://azure.microsoft.com/pricing/details/cognitive-services/openai-service/) beyond token-based fees.

> Sessions: 1-hour active / 30-min idle timeout. Each conversation = separate billable session.

> ⚠️ When Code Interpreter is used through a toolbox in a **hosted agent**, user isolation isn't supported — all users in the same project share one container context.

## Prompt-agent SDK class

`CodeInterpreterTool` — see [tool-mcp.md](tool-mcp.md) for the general prompt-agent tool-wiring pattern; Code Interpreter takes no constructor arguments.

## Toolbox shape

```json
{ "type": "code_interpreter" }
```

No other fields. Only one `code_interpreter` per toolbox version (unnamed tool).

## References

- [Code Interpreter tool documentation](https://learn.microsoft.com/azure/foundry/agents/how-to/tools/code-interpreter)
- [agent-tools.md](agent-tools.md) — tool index
- [toolbox-reference.md](toolbox-reference.md) — endpoint, auth, and MCP protocol details

# Tool — Fabric IQ (preview)

Connect an agent to Microsoft Fabric data — Ontology, Fabric data agents, and Power BI semantic models — through **Fabric IQ**. The agent delegates natural-language questions; Fabric IQ runs them against the enterprise ontology (NL2Ontology) and returns synthesized answers under the signed-in user's Fabric permissions.

## Toolbox shape

```json
{
  "type": "fabric_iq_preview",
  "project_connection_id": "<fabriciq-connection-name>",
  "server_label": "<short-lowercase-label>",
  "server_url": "https://<host>/v1/mcp/..."
}
```

## `server_url` by Fabric item type

| Fabric item | `server_url` pattern | Supported auth |
|---|---|---|
| Ontology | `https://{host}/v1/mcp/dataPlane/workspaces/{workspaceId}/items/{itemId}/ontologyEndpoint` | BYO Entra app only |
| Fabric data agent | `https://{host}/v1/mcp/workspaces/{workspaceId}/dataagents/{dataAgentId}/agent` | BYO Entra app *or* managed OAuth |
| Power BI semantic model | `https://{host}/v1/mcp/fabricaihub/integrations/m365` | BYO Entra app *or* managed OAuth |

## Requirements

- Microsoft Fabric license for both the developer and every calling end-user.
- For Ontology / Power BI: Entra app with delegated Power BI permissions `Item.Execute.All` + `Item.Read.All`; tenant admin consent required. For Data Agent: `DataAgent.Execute.All`.
- Each Fabric item must be **published** before it can be consumed through Fabric IQ.
- VNet integration is **not** supported.
- Tip: for Power BI semantic models, use latest models — measure/hierarchy reasoning benefits significantly.

## References

For the full Entra app setup, connection-creation walkthrough, and troubleshooting, see [Fabric IQ tool documentation](https://learn.microsoft.com/azure/foundry/agents/how-to/tools/fabric-iq).

- [agent-tools.md](agent-tools.md) — tool index
- [foundry-tool-catalog.md](foundry-tool-catalog.md) — connection shape for Fabric IQ

# File Search Tool

Enables agents to search through uploaded files using semantic and keyword search from vector stores. Supports a wide range of file formats including PDF, Markdown, Word, and more.

> ⚠️ **Important:** Before creating an agent with file search, you **must** read the official documentation linked in the References section to understand prerequisites, supported file types, and vector store setup.

## Prerequisites

- A [basic or standard agent environment](https://learn.microsoft.com/azure/ai-foundry/agents/environment-setup)
- A **vector store** must be created before the agent — the `file_search` tool requires `vector_store_ids`
- Files must be uploaded to the vector store before the agent can search them

## Key Concepts

| Concept | Description |
|---------|-------------|
| **Vector Store** | A container that indexes uploaded files for semantic search. Must be created first. |
| **vector_store_ids** | Required parameter on the `file_search` tool — references the vector store(s) to search. |
| **File upload** | Files are uploaded to the project, then attached to a vector store for indexing. |

## Setup Workflow

```
1. Create a vector store (REST API: POST /vector_stores)
   │
   ▼
2. (Optional) Upload files and attach to vector store
   │
   ▼
3. Create agent with file_search tool referencing the vector_store_ids
   │
   ▼
4. Agent can now search files in the vector store
```

> ⚠️ **Warning:** Creating an agent with `file_search` without providing `vector_store_ids` will fail with a `400 BadRequest` error: `required: Required properties ["vector_store_ids"] are not present`.

## REST API Notes

When creating vector stores via `az rest`:

| Parameter | Value |
|-----------|-------|
| **Endpoint** | `https://<resource>.services.ai.azure.com/api/projects/<project>/vector_stores` |
| **API version** | `v1` |
| **Auth resource** | `https://ai.azure.com` |

## Troubleshooting

| Error | Cause | Fix |
|-------|-------|-----|
| `vector_store_ids` not present | Agent created without vector store | Create a vector store first, then pass its ID |
| 401 Unauthorized | Wrong auth resource for REST API | Use `--resource "https://ai.azure.com"` with `az rest` |
| Bad API version | Using ARM-style API version | Use `api-version=v1` for the data-plane vector store API |
| No search results | Vector store is empty | Upload files to the vector store before querying |

## References

- [File Search tool documentation](https://learn.microsoft.com/azure/ai-foundry/agents/how-to/tools/file-search?view=foundry&pivots=python)
- [Tool Catalog](https://learn.microsoft.com/azure/ai-foundry/agents/concepts/tool-catalog?view=foundry)

# Tool — Function Calling (client-side)

Define custom functions the agent can invoke. Your app executes the function and returns results. Runs expire 10 minutes after creation — return tool outputs promptly.

> **Security:** Treat tool arguments as untrusted input. Don't pass secrets in tool output. Use `strict=True` for schema validation.

> **Not available via toolbox** — function calling executes in the client process, so it's declared on the prompt agent, not in a toolbox version.

## Prompt-agent SDK class

`FunctionTool` — wraps a Python callable; the SDK introspects its signature and docstring to build the schema sent to the model.

## References

- [Function Calling tool documentation](https://learn.microsoft.com/azure/foundry/agents/how-to/tools/function-calling)
- [agent-tools.md](agent-tools.md) — tool index

# MCP Tool (Model Context Protocol)

Connect agents to remote MCP servers to extend capabilities with external tools and data sources. MCP is an open standard for LLM tool integration.

## Prerequisites

- A remote MCP server endpoint (e.g., `https://api.githubcopilot.com/mcp`)
- For authenticated servers: a [project connection](../../../project/connections.md) storing credentials
- RBAC: **Contributor** or **Owner** role on the Foundry project

## Authenticated Server Connections

For authenticated MCP servers, create an `api_key` project connection to store credentials. Unauthenticated servers (public endpoints) don't need a connection — omit `project_connection_id`.

See [Project Connections](../../../project/connections.md) for connection management via Foundry MCP tools.

## MCPTool Parameters

| Parameter | Required | Description |
|-----------|----------|-------------|
| `server_label` | Yes | Unique label for this MCP server within the agent |
| `server_url` | Yes | Remote MCP server endpoint URL |
| `require_approval` | No | `"always"` (default), `"never"`, or `{"never": ["tool1"]}` / `{"always": ["tool1"]}` |
| `allowed_tools` | No | List of specific tools to enable (default: all) |
| `project_connection_id` | No | Connection ID for authenticated servers |

## Approval Workflow

1. Agent sends request → MCP server returns tool calls
2. Response contains `mcp_approval_request` items
3. Your code reviews tool name + arguments
4. Submit `McpApprovalResponse` with `approve=True/False`
5. Agent completes work using approved tool results

> **Best practice:** Always use `require_approval="always"` unless you fully trust the MCP server. Use `allowed_tools` to restrict which tools the agent can access.

## Hosting Local MCP Servers

Agent Service only accepts **remote** MCP endpoints. To use a local server, deploy it to:

| Platform | Transport | Notes |
|----------|-----------|-------|
| [Azure Container Apps](https://github.com/Azure-Samples/mcp-container-ts) | HTTP POST/GET | Any language, container rebuild needed |
| [Azure Functions](https://github.com/Azure-Samples/mcp-sdk-functions-hosting-python) | HTTP streamable | Python/Node/.NET/Java, key-based auth |

## Known Limitations

- **100-second timeout** for non-streaming MCP tool calls
- **Identity passthrough not supported in Teams** — agents published to Teams use project managed identity
- **Network-secured Foundry** can't use private MCP servers in same vNET — only public endpoints

## Troubleshooting

| Error | Cause | Fix |
|-------|-------|-----|
| `Invalid tool schema` | `anyOf`/`allOf` in MCP server definition | Update MCP server schema to use simple types |
| `Unauthorized` / `Forbidden` | Wrong credentials in connection | Verify connection credentials match server requirements |
| Model never calls MCP tool | Misconfigured server_label/url | Check `server_label`, `server_url`, `allowed_tools` values |
| Agent stalls after approval | Missing `previous_response_id` | Include `previous_response_id` in follow-up request |
| Timeout | Server takes >100s | Optimize server-side logic or break into smaller operations |

## References

- [MCP tool documentation](https://learn.microsoft.com/azure/ai-foundry/agents/how-to/tools/mcp?view=foundry)
- [Tool Catalog](https://learn.microsoft.com/azure/ai-foundry/agents/concepts/tool-catalog?view=foundry)
- [Project Connections](../../../project/connections.md)

# Agent Memory

Managed long-term memory for Foundry agents. Enables agent continuity across sessions, devices, and workflows. Agents retain user preferences, conversation history, and deliver personalized experiences. Memory is stored in your project's owned storage.

## Prerequisites

- A [Foundry project](https://learn.microsoft.com/azure/ai-foundry/how-to/create-projects) with authorization configured
- A **chat model deployment** (e.g., `gpt-5.2`)
- An **embedding model deployment** (e.g., `text-embedding-3-small`) — see [Check Embedding Model](#check-embedding-model) below
- Python packages: `pip install azure-ai-projects azure-identity`

### Check Embedding Model

An embedding model is **required** before enabling memory. Check if one is already deployed:

Use `foundry_models_list` MCP tool to list all deployments and look for an embedding model (e.g., `text-embedding-3-small`, `text-embedding-3-large`, `text-embedding-ada-002`).

| Result | Action |
|--------|--------|
| ✅ Embedding model found | Note the deployment name and proceed |
| ❌ No embedding model | Deploy one before enabling memory — see below |

### Deploy Embedding Model

If no embedding model exists, use `foundry_models_deploy` MCP tool with:
- `deploymentName`: `text-embedding-3-small` (or preferred name)
- `modelName`: `text-embedding-3-small`
- `modelFormat`: `OpenAI`

## Authorization and Permissions

| Role | Scope | Purpose |
|------|-------|---------|
| **Foundry User** | AI Services resource | Assigned to project managed identity |
| **System-assigned managed identity** | Project | Must be enabled on the project |

**Setup steps:**
1. In Azure portal → project → **Resource Management** → **Identity** → enable system-assigned managed identity
2. On the AI Services resource → **Access control (IAM)** → assign **Foundry User** to the project managed identity

## Workflow

```
User wants agent memory
    │
    ▼
Step 1: Check for embedding model deployment
    │  ├─ ✅ Found → Continue
    │  └─ ❌ Not found → Deploy one (ask user)
    │
    ▼
Step 2: Create memory store
    │
    ▼
Step 3: Attach memory tool to agent
    │
    ▼
Step 4: Test with conversation
```

## Key Concepts

### Memory Store Options

| Option | Description |
|--------|-------------|
| `chat_summary_enabled` | Summarize conversations for memory |
| `user_profile_enabled` | Build and maintain user profile |
| `user_profile_details` | Control what data gets stored (e.g., `"Avoid sensitive data such as age, financials, location, credentials"`) |

> 💡 **Tip:** Use `user_profile_details` to control what the agent stores — e.g., `"flight carrier preference and dietary restrictions"` for a travel agent, or exclude sensitive data.

### Scope

The `scope` parameter partitions memory per user:

| Scope Value | Behavior |
|-------------|----------|
| `{{$userId}}` | Auto-extracts TID+OID from auth token (recommended) |
| `"user_123"` | Static identifier — you manage user mapping |

### Memory Store Operations

| Operation | Description |
|-----------|-------------|
| Create | Initialize a memory store with chat/embedding models and options |
| List | List all memory stores in the project |
| Update | Update memory store description or configuration |
| Delete scope | Delete memories for a specific user scope |
| Delete store | Delete entire memory store (irreversible — all scopes lost) |

> ⚠️ **Warning:** Deleting a memory store removes all memories across all scopes. Agents with attached memory stores lose access to historical context.

## Troubleshooting

| Issue | Cause | Resolution |
|-------|-------|------------|
| Auth/authorization error | Identity or managed identity lacks required roles | Verify roles in Authorization section; refresh access token for REST |
| Memories don't appear after conversation | Updates are debounced or still processing | Increase wait time or call update API with `update_delay=0` |
| Memory search returns no results | Scope mismatch between update and search | Use same scope value for storing and retrieving memories |
| Agent response ignores stored memory | Agent not configured with memory search tool | Confirm agent definition includes `MemorySearchTool` with correct store name |
| No embedding model available | Embedding deployment missing | Deploy an embedding model — see Check Embedding Model section |

## References

- [Memory tool documentation](https://learn.microsoft.com/azure/ai-foundry/agents/how-to/memory-usage?view=foundry)
- [Memory Concepts](https://learn.microsoft.com/azure/ai-foundry/agents/concepts/what-is-memory)
- [Tool Catalog](https://learn.microsoft.com/azure/ai-foundry/agents/concepts/tool-catalog?view=foundry)
- [Python Samples](https://github.com/Azure/azure-sdk-for-python/tree/main/sdk/ai/azure-ai-projects/samples/memories)

# Tool — OpenAPI

Expose a REST API to the agent by attaching its OpenAPI 3.x spec. The platform parses the spec and synthesizes one tool per operation.

## Toolbox shape (anonymous)

```json
{
  "type": "openapi",
  "openapi": {
    "spec": { /* inlined OpenAPI 3.x document */ },
    "auth": { "type": "anonymous" }
  }
}
```

## `auth.type` values

- **`anonymous`** — no credentials sent.
- **`connection`** with `project_connection_id` — Foundry attaches a static API key (or OAuth tokens) from the named project connection. **`project_connection_id` is required only here.**
- **`managed_identity`** with `audience` — the project's managed identity calls the target API. **No `project_connection_id` is required**; Foundry uses the project MI and acquires a token for the supplied `audience` (the target service's resource URI). You must grant the project MI the appropriate RBAC role on the target service or the agent receives `401 Unauthorized`.

## Multi-entry rules

Multiple `openapi` entries are allowed in one toolbox **only if** each entry's spec defines a distinct `info.title` (the title is the implicit identifier). See [toolbox-reference.md § Multi-Tool Toolbox Constraint](toolbox-reference.md#multi-tool-toolbox-constraint).

## References

- [OpenAPI tool documentation](https://learn.microsoft.com/azure/foundry/agents/how-to/tools/openapi)
- [agent-tools.md](agent-tools.md) — tool index
- [foundry-tool-catalog.md](foundry-tool-catalog.md) — project connections for the `connection` auth path

# Tool — Tool Search (preview)

For toolboxes containing many tools, replace the full tool list passed to the model with two meta-tools — `tool_search` (natural-language discovery, returns matching tools per query) and `call_tool` (invoke any discovered tool by name) — so context cost stays flat regardless of toolbox size.

## Toolbox shape

```json
{ "type": "toolbox_search_preview" }
```

## Behavior

- `toolbox_search_preview` is a **configuration directive** — it doesn't appear in `tools/list` itself and doesn't count toward the unnamed-tool-per-type limit.
- All other toolbox tools are **hidden** from the initial `tools/list` and are returned only by `tool_search` calls (or by per-user auto-pinning of hot tools).
- Pin specific tools or add search-only keywords via `tool_configs.{tool_name}`:

  ```json
  {
    "type": "mcp",
    "server_label": "analytics",
    "server_url": "https://db-mcp.internal/sse",
    "tool_configs": {
      "execute_query": { "pin": true, "additional_search_text": "SQL analytics reporting dashboard" },
      "*":             { "additional_search_text": "data warehouse queries" }
    }
  }
  ```

  Use `"*"` as the key to apply settings to all tools in that entry.
- `additional_search_text` is used only for search ranking — it's never exposed to the model in the tool schema.
- Tool **descriptions drive match quality**: every MCP tool should have a clear `description`, or `tool_search` won't find it.
- Recommendation: add an instruction in the system prompt telling the model to call `tool_search` when a needed capability isn't in its current tool list.

## References

For full fields, pinning recipes, the verify-with-`tool_search` flow, and best practices, see [Tool Search tool documentation](https://learn.microsoft.com/azure/foundry/agents/how-to/tools/tool-search).

- [agent-tools.md](agent-tools.md) — tool index
- [use-toolbox-in-hosted-agent.md § Recommendation: enable Tool Search](use-toolbox-in-hosted-agent.md#-recommendation-enable-tool-search)

# Web Search Tool (Preview)

Enables agents to retrieve and ground responses with real-time public web information before generating output. Returns up-to-date answers with inline URL citations. This is the **default tool for web search** — no external resource or connection setup required.

> ⚠️ **Warning:** For Bing Grounding or Bing Custom Search (which require a separate Bing resource and project connection), see [tool-bing-grounding.md](tool-bing-grounding.md). Only use those when explicitly requested.

## Important Disclosures

- Web Search (preview) uses Grounding with Bing Search and Grounding with Bing Custom Search, which are [First Party Consumption Services](https://www.microsoft.com/licensing/terms/product/Glossary/EAEAS) governed by [Grounding with Bing terms of use](https://www.microsoft.com/bing/apis/grounding-legal-enterprise) and the [Microsoft Privacy Statement](https://go.microsoft.com/fwlink/?LinkId=521839&clcid=0x409).
- The [Data Protection Addendum](https://aka.ms/dpa) **does not apply** to data sent to Grounding with Bing Search and Grounding with Bing Custom Search.
- Data transfers occur **outside compliance and geographic boundaries**.
- Usage incurs costs — see [pricing](https://www.microsoft.com/bing/apis/grounding-pricing).

## Prerequisites

- A [basic or standard agent environment](https://learn.microsoft.com/azure/ai-foundry/agents/environment-setup)
- Azure credentials configured (e.g., `DefaultAzureCredential`)

## Setup

No external resource or project connection is required. The web search tool works out of the box when added to an agent definition.

## Configuration Options

| Parameter | Description | Default |
|-----------|-------------|---------|
| `user_location` | Approximate location (country/region/city) for localized results | None |
| `search_context_size` | Context window space for search: `low`, `medium`, `high` | `medium` |

## Administrator Control

Admins can enable or disable web search at the subscription level via Azure CLI. Requires Owner or Contributor access.

- **Disable:** `az feature register --name OpenAI.BlockedTools.web_search --namespace Microsoft.CognitiveServices --subscription "<subscription-id>"`
- **Enable:** `az feature unregister --name OpenAI.BlockedTools.web_search --namespace Microsoft.CognitiveServices --subscription "<subscription-id>"`

## Security Considerations

- Treat web search results as **untrusted input**. Validate before use in downstream systems.
- Avoid sending secrets or sensitive data in prompts forwarded to external services.

## Troubleshooting

| Issue | Cause | Resolution |
|-------|-------|------------|
| No citations appear | Model didn't determine web search was needed | Update instructions to explicitly allow web search; ask queries requiring current info |
| Requests fail after enabling | Web search disabled at subscription level | Ask admin to enable — see Administrator Control above |
| Authentication errors (REST) | Bearer token missing, expired, or insufficient | Refresh token; confirm project/agent access |
| Outdated results | Content not recently indexed by Bing | Refine query to request most recent info |
| No results for specific topics | Query too narrow | Broaden query; niche topics may have limited coverage |
| Rate limiting (429) | Too many requests | Implement exponential backoff; space out requests |

## References

- [Web Search tool documentation](https://learn.microsoft.com/azure/ai-foundry/agents/how-to/tools/web-search?view=foundry)
- [Tool Catalog](https://learn.microsoft.com/azure/ai-foundry/agents/concepts/tool-catalog?view=foundry)
- [Bing Pricing](https://www.microsoft.com/bing/apis/grounding-pricing)

# Tool — Work IQ (preview)

Connect an agent to the user's Microsoft 365 work context — email, meetings, files, chats — through **Work IQ**. Work IQ runs as an A2A peer; every request runs in the context of the signed-in user and honors all Microsoft 365 permissions and sensitivity labels.

> 🚦 **Toolbox creation gate:** before creating a toolbox/connection, you MUST read the boundary rules in [create-hosted.md → Toolbox creation boundary](../create-hosted.md#toolbox-creation-boundary) and follow them, then continue with the rest of this file.

## Toolbox shape

```json
{
  "type": "work_iq_preview",
  "project_connection_id": "<workiq-connection-name>"
}
```

## Requirements

- A `RemoteA2A` project connection targeting `https://workiq.svc.cloud.microsoft/a2a/`, `authType=OAuth2`, **BYO Entra app only** (no managed OAuth).
- Scopes: `api://workiq.svc.cloud.microsoft/WorkIQAgent.Ask` + `offline_access`. A **Global Administrator** must grant tenant-wide admin consent for `WorkIQAgent.Ask` (Work IQ app ID `fdcc1f02-fc51-4226-8753-f668596af7f7`).
- Each calling end-user must hold a **Microsoft 365 Copilot license**.
- The Work IQ service principal must be pre-provisioned in the tenant (one-time, via Graph Explorer); see the public doc.
- VNet integration is **not** supported — the Foundry project must not use a VNet-restricted endpoint.

## References

For the full Entra app setup, ARM connection-creation payload (`category: RemoteA2A`), and troubleshooting, see [Work IQ tool documentation](https://learn.microsoft.com/azure/foundry/agents/how-to/tools/work-iq).

- [agent-tools.md](agent-tools.md) — tool index
- [foundry-tool-catalog.md](foundry-tool-catalog.md) — RemoteA2A connection shape

# Toolbox Reference

Endpoint format, MCP protocol details, authentication, OAuth consent handling, endpoint testing, citation pattern, and troubleshooting for Foundry Toolboxes.

## Endpoint Format

The toolbox MCP endpoint is constructed from the **project endpoint** + **toolbox name**:

| Endpoint | URL |
|----------|-----|
| Latest version (default) | `{project_endpoint}/toolboxes/{toolbox_name}/mcp?api-version=v1` |
| Specific version | `{project_endpoint}/toolboxes/{toolbox_name}/versions/{version}/mcp?api-version=v1` |

- **Project endpoint** format: `https://<account>.services.ai.azure.com/api/projects/<project>`
- The latest-version endpoint always serves the toolbox's `default_version`.
- Use the specific-version endpoint to test a version before promoting it.
- `?api-version=v1` query parameter is **required** — requests without it return HTTP 400.

### Agent env contract

Hosted agents read the MCP endpoint from a single environment variable. The **canonical** name is **`TOOLBOX_ENDPOINT`** — use it in all new code and `.env` files:

```
# Latest version (recommended for prod):
TOOLBOX_ENDPOINT=https://{host}/api/projects/{project}/toolboxes/{toolbox_name}/mcp?api-version=v1

# Pinned to a specific version (recommended for testing a new version before promoting):
TOOLBOX_ENDPOINT=https://{host}/api/projects/{project}/toolboxes/{toolbox_name}/versions/{version}/mcp?api-version=v1
```

> ⚠️ **Don't use `FOUNDRY_TOOLBOX_ENDPOINT` in new code.** The Foundry platform **reserves** all environment variables prefixed `FOUNDRY_` and may silently overwrite user-defined values at runtime. Always use a name without the `FOUNDRY_` prefix (e.g. `TOOLBOX_ENDPOINT` or `TOOLBOX_MCP_ENDPOINT`). Some older samples still reference `FOUNDRY_TOOLBOX_ENDPOINT` — treat that as **deprecated/legacy** and only fall back to it when maintaining a sample that already wires it.

## MCP Protocol

Toolboxes use **Model Context Protocol (MCP)** — JSON-RPC 2.0 over HTTP POST:

- **`initialize`** — Optional MCP handshake. The toolbox endpoint is effectively **stateless**: it does **not** return an `mcp-session-id` header, and `tools/list` / `tools/call` work without first calling `initialize` or passing any session header.
- **`tools/list`** — Returns all available tools with names, descriptions, and input schemas.
- **`tools/call`** — Invokes a tool with arguments and returns structured results.

> `prompts/list` is **not supported** by the toolbox endpoint. Always pass `load_prompts=False` to MCP client constructors.

### Tool naming

- **MCP-sourced tools** (`type: mcp`) are exposed as `{server_label}___{tool_name}` — joined by **three underscores** (e.g. `myserver___get_info`). Call them with the prefixed name in `tools/call`.
- **All other tool types** (`web_search`, `file_search`, `azure_ai_search`, `code_interpreter`, `openapi`, `a2a_preview`, `work_iq_preview`, `fabric_iq_preview`, …) use the value of the entry's `name` field, or the default tool name if `name` is unset.
- **Tool Search** injects two platform meta-tools whose names are always `tool_search` and `call_tool`.

Each tool returned by `tools/list` includes a `_meta.tool_configuration` block with at least the `type`, plus type-specific fields (e.g. `server_label`, `server_url`, `require_approval` for MCP).

## Authentication

- **Agent → Toolbox:** Azure AD bearer token with scope `https://ai.azure.com/.default`, refreshed on every request.
- **Toolbox → External Services:** Managed by the platform via project connections (API keys, OAuth, managed identity). See [foundry-tool-catalog.md](foundry-tool-catalog.md) for the connection shapes that back each tool type.

> ⚠️ Do **not** use scope `https://cognitiveservices.azure.com/.default`. The toolbox MCP endpoint rejects it with HTTP 401.

> 💡 **If you're hitting OAuth errors (e.g. `CONSENT_REQUIRED`, ARA-style "authentication required" errors, or 401s) when calling an MCP server directly from your agent**, switch to wiring the same MCP server into a toolbox. The toolbox handles the full OAuth flow — bearer-token acquisition, refresh, consent discovery, and per-user token passthrough — so your agent only ever talks to the toolbox MCP endpoint with a standard `https://ai.azure.com/.default` token. This is the recommended path for any OAuth-based MCP server in a hosted agent.

## OAuth Consent Handling

When a toolbox includes an OAuth-based MCP connection (e.g., GitHub OAuth), the **first** call from a new user surfaces a consent requirement. The toolbox wraps per-source failures in a JSON-RPC error with outer **code `-32006`** (`tools/list failed for N tool source(s)…`); the failing source's nested error carries the **string** code `"CONSENT_REQUIRED"`, and its `message` is the consent URL. This can surface on `tools/list`, `initialize`, or `tools/call` depending on when the server discovers the missing grant.

**Agent code must handle this:**

1. On a `-32006` error, extract the embedded JSON from the outer `message`. **The message is not directly JSON-parseable** — it begins with a human-readable prefix (`tools/list failed for N tool source(s), succeeded for M tool source(s) `) followed by a `{"errors":[...]}` payload. Locate the first `{` and parse from there (do **not** call `JSON.parse` on the whole `message`):

   ```python
   msg = err["message"]
   payload = json.loads(msg[msg.index("{"):])   # slice off the prefix first
   ```

2. Detect the nested `"code":"CONSENT_REQUIRED"` in `payload["errors"][i]["error"]` and read the consent URL from that nested `message`.
3. Log the URL and surface it to the user (e.g., print to stdout or return in the agent response).
4. After the user completes the OAuth flow in a browser, retry the call — subsequent calls succeed without re-prompting.

Example error shape (as actually returned — note the prefix text before the JSON):

```json
{
  "jsonrpc": "2.0",
  "id": 1,
  "error": {
    "code": -32006,
    "message": "tools/list failed for 1 tool source(s), succeeded for 0 tool source(s) {\"errors\":[{\"name\":\"GitHub\",\"type\":\"mcp\",\"error\":{\"code\":\"CONSENT_REQUIRED\",\"message\":\"https://logic-apis-<region>.consent.azure-apim.net/login?data=...\"}}]}"
  }
}
```

> This is a one-time flow per user per OAuth connection in a project. The agent should not silently swallow this error.

## Multi-Tool Toolbox Constraint

A single toolbox can combine multiple tools, but **at most one tool per unnamed tool type**. Tools like `web_search`, `file_search`, `azure_ai_search`, and `code_interpreter` have no identifier; to include more than one instance of the same type, set a unique `name` on each instance. MCP tools must each have a unique `server_label`.

If you include two unnamed tools of the same type (or two MCP tools with the same `server_label`), the API returns:

```
400 invalid_payload: Multiple tools without identifiers found...
```

Valid combinations include:

- `file_search` + one or more `mcp` (each with unique `server_label`)
- `web_search` + one or more `mcp`
- `azure_ai_search` + one or more `mcp`
- `toolbox_search_preview` (the Tool Search directive — doesn't count toward the limit) + any other tools

## Azure AI Search Citation Pattern

When calling an `azure_ai_search` tool through the toolbox MCP endpoint, citation metadata is returned under `result.structuredContent.documents[]` — **not** in a separate `citations` array. Treat each document as one citation:

| Field | Meaning |
|-------|---------|
| `title` | Citation display text |
| `url` | Source link |
| `id` | Source identifier |
| `score` | Retrieval relevance score |
| `knowledgeSourceIndex` | Source grouping / index |

Verification checklist:

1. `tools/list` returns the tool name `azure_ai_search`.
2. `tools/call` succeeds with a `query` argument.
3. `result.structuredContent.documents` is present and non-empty.
4. At least one document has both `title` and `url`.

For File Search and Web Search citation patterns (under `result.content[].resource._meta` and `..._meta.annotations[]` respectively), see the public [Toolbox docs](https://learn.microsoft.com/azure/foundry/agents/how-to/tools/toolbox).

## Testing the Toolbox Endpoint

Before running the full agent, verify the toolbox MCP endpoint works end-to-end. Use `az login` for authentication, then test the three MCP operations in order:

**1. Get a bearer token:**

```bash
TOKEN=$(az account get-access-token --resource https://ai.azure.com --query accessToken -o tsv)
TOOLBOX_URL="https://<account>.services.ai.azure.com/api/projects/<project>/toolboxes/<name>/mcp?api-version=v1"
```

**2. (Optional) Initialize MCP session:**

The toolbox endpoint is stateless, so this step is not required — `tools/list` and `tools/call` work without it. Run it only to confirm the handshake:

```bash
curl -sS -X POST "$TOOLBOX_URL" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-03-26","capabilities":{},"clientInfo":{"name":"debug","version":"1.0.0"}}}' \
  -D - | head -20
```

No `mcp-session-id` header is returned, and none is needed on later calls.

**3. List tools:**

```bash
curl -sS -X POST "$TOOLBOX_URL" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","id":2,"method":"tools/list","params":{}}' | jq .
```

Checklist:

- Response contains `result.tools[]` with `len > 0`
- Each tool has `name`, `description`, and `inputSchema` with a `properties` field
- MCP tool names for remote servers are prefixed with `server_label` joined by three underscores (e.g., `myserver___get_info`)
- All other tool types use the entry's `name` field value (or the default tool name)

**4. Call a tool (optional):**

```bash
curl -sS -X POST "$TOOLBOX_URL" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","id":3,"method":"tools/call","params":{"name":"<tool_name>","arguments":{"query":"test"}}}' | jq .
```

## Troubleshooting

| Error | Cause | Resolution |
|-------|-------|------------|
| `CONSENT_REQUIRED` (nested string code inside an outer `-32006` error) | OAuth MCP connection needs user consent | Parse the consent URL from the nested error `message`, open it in a browser, complete OAuth, retry |
| 401 on MCP calls | Expired token or wrong scope | Use scope `https://ai.azure.com/.default` (not `cognitiveservices`) and refresh token on every request |
| OAuth/ARA errors when calling MCP directly from agent | Direct MCP wiring without toolbox token passthrough | Wire the MCP server into a toolbox and call the toolbox endpoint instead — Foundry handles consent + refresh |
| 400 `invalid_payload: Multiple tools without identifiers found` | Two unnamed tools of the same type (or duplicate `server_label`) in one toolbox | Keep at most one unnamed tool per type; give each MCP tool a unique `server_label` |
| `tools/list` returns 0 tools | Toolbox version still provisioning, or tool type not yet available in the region | Wait ~10s and retry; try a different region |
| `tools/list` returns 0 tools for MCP/A2A only | Invalid or missing connection credentials | Verify `project_connection_id` exists and credentials are correct; for MI auth, check RBAC on the target service |
| `tools/list` returns 0 tools for OpenAPI only | Invalid OpenAPI spec (malformed paths, missing operationIds) | Validate the spec against OpenAPI 3.0/3.1; for MI auth, also verify RBAC |
| Tool not found on `tools/call` | Missing `server_label___` prefix for MCP-sourced tools | Call as `{server_label}___{tool_name}` (three underscores) |
| 500 on `prompts/list` | Not supported by toolbox endpoint | Pass `load_prompts=False` to MCP client constructor |
| 500 on `send_ping()` (MAF `MCPStreamableHTTPTool._ensure_connected`) | Toolbox MCP server doesn't implement `ping` | Disable the ping check or override with a no-op |
| 500 with non-streaming `tools/call` | Non-streaming not supported | Always use `stream=True` for toolbox MCP tools |
| 400 missing `api-version` | Query string dropped | Append `?api-version=v1` to every toolbox URL |
| Environment variable silently overwritten at runtime | Foundry reserves `FOUNDRY_`-prefixed env vars | Rename to a non-`FOUNDRY_` name (e.g. `TOOLBOX_ENDPOINT`) |
| 403 on `POST /toolboxes` or `PUT .../connections/...` | Caller lacks `Foundry User` (or `Azure AI Developer` / `Cognitive Services Contributor`) on the project | Grant the role at the project scope |

# Tools and Toolboxes (azd ai)

How to attach tools (web search, Azure AI Search, MCP, A2A) to a hosted agent using `azd ai toolbox` and `azd ai connection`.

A **toolbox** is a curated bundle of connection-backed tools that Foundry exposes as a single MCP-compatible endpoint. The agent connects to one URL and discovers every tool inside. `azd deploy` does NOT auto-create toolboxes -- you drive the lifecycle explicitly.

> 🚦 **Toolbox creation gate:** before creating a toolbox/connection, you MUST read the boundary rules in [create-hosted.md → Toolbox creation boundary](../create-hosted.md#toolbox-creation-boundary) and follow them, then continue with the rest of this file.

## Install the extension once

```bash
azd extension install azure.ai.toolboxes
```

## The flow (every recipe)

1. Create the **connection** (`azd ai connection create ...`).
2. Create the **toolbox** (`azd ai toolbox create`) or add tools to an existing one (`azd ai toolbox connection add`).
3. If you added to an existing toolbox, **promote the new version** (`azd ai toolbox publish <name> <version>`) — `create` publishes its first version automatically, but later mutations do not.
4. Read the endpoint (`azd ai toolbox show <name> --output json`).
5. `azd env set TOOLBOX_<NAME>_MCP_ENDPOINT "<endpoint>"`.
6. Reference it in the agent service's `environmentVariables` in `azure.yaml`.
7. `azd deploy`.

## Env var naming convention

Uppercase the toolbox name, collapse non-alphanumeric to `_`, prefix `TOOLBOX_`, suffix `_MCP_ENDPOINT`. Examples: `agent-tools` -> `TOOLBOX_AGENT_TOOLS_MCP_ENDPOINT`, `agent.tools.v2` -> `TOOLBOX_AGENT_TOOLS_V2_MCP_ENDPOINT`.

## Endpoint URL shapes

- `{project}/toolboxes/{name}/versions/{version}/mcp?api-version=v1` -- version-pinned. What `azd ai toolbox show` returns.
- `{project}/toolboxes/{name}/mcp?api-version=v1` -- default version (consumer). Always serves `default_version`.

To auto-pick up new default versions without redeploying, drop the `/versions/<ver>` segment and store the consumer URL.

## CLI surface

| Command | What it does |
|---------|--------------|
| `azd ai toolbox create <name> --from-file <path>` | Create toolbox + its first version. File must list at least one connection, skill, or tool. |
| `azd ai toolbox connection add <toolbox> <connection> [--index ...] [--instance-name ...]` | Attach one; creates a new version (default unchanged). |
| `azd ai toolbox connection add <toolbox> --from-file <path>` | Attach many in one call; ONE new version (default unchanged). |
| `azd ai toolbox connection remove <toolbox> <connection>` | Detach; creates a new version (default unchanged). Refuses to leave zero tools. |
| `azd ai toolbox show <name> [--version <ver>]` | Show toolbox + MCP endpoint URL. |
| `azd ai toolbox list` | List toolboxes. |
| `azd ai toolbox versions list <toolbox>` | List versions. |
| `azd ai toolbox publish <name> <version>` | Promote a version to default (also used to roll back). |
| `azd ai toolbox delete <name> [--version <ver>] [--force]` | Delete toolbox or one version. |

Every mutation publishes a new immutable version but does **not** change the default; run `azd ai toolbox publish <name> <version>` to promote one.

## `--from-file` shape

```yaml
description: research toolbox    # only on `create`
connections:
  - name: my-mcp                 # RemoteTool
  - name: my-search              # CognitiveSearch -- needs index
    index: products
  - name: my-bing                # GroundingWithCustomSearch -- needs instance_name
    instance_name: docs-config
  - name: my-a2a                 # RemoteA2A
```

## Recipe: GitHub MCP

```bash
# 1. Connection
azd ai connection create github-mcp-conn \
  --kind remote-tool \
  --target https://api.githubcopilot.com/mcp \
  --auth-type custom-keys \
  --custom-key Authorization="Bearer ghp_xxx..."

# 2. Toolbox (initial create needs a file; otherwise use `connection add`)
cat > tools.json <<EOF
{ "description": "GitHub MCP", "connections": [{ "name": "github-mcp-conn" }] }
EOF
azd ai toolbox create agent-tools --from-file tools.json

# 3. Wire the env var
ENDPOINT=$(azd ai toolbox show agent-tools --output json | jq -r .endpoint)
azd env set TOOLBOX_AGENT_TOOLS_MCP_ENDPOINT "$ENDPOINT"
```

Add the env var to the agent service's `environmentVariables` in `azure.yaml`:

```yaml
environmentVariables:
  - name: TOOLBOX_AGENT_TOOLS_MCP_ENDPOINT
    value: ${TOOLBOX_AGENT_TOOLS_MCP_ENDPOINT}
```

Then `azd deploy`.

## Recipe: Azure AI Search RAG

```bash
azd ai connection create my-search-conn \
  --kind cognitive-search \
  --target https://my-search.search.windows.net/ \
  --auth-type api-key --key "<search-admin-key>"

azd ai toolbox connection add agent-tools my-search-conn --index contoso-outdoors
```

For multiple indexes, add multiple entries with different `index` values.

## Recipe: A2A peer agent

```bash
azd ai connection create peer-agent-conn \
  --kind remote-a2a \
  --target https://other-agent.foundry-account.westus2.azure.com/ \
  --auth-type none

azd ai toolbox connection add agent-tools peer-agent-conn
```

For authenticated peers, use `--auth-type project-managed-identity --audience https://ai.azure.com/.default`.

## Recipe: multi-tool toolbox in one call

```yaml
# tools.yaml
description: "GitHub MCP + AI Search + A2A peer."
connections:
  - name: github-mcp-conn
  - name: my-search-conn
    index: contoso-outdoors
  - name: peer-agent-conn
```

```bash
azd ai toolbox create agent-tools --from-file tools.yaml
# OR (existing toolbox): azd ai toolbox connection add agent-tools --from-file tools.yaml
#   then promote it: azd ai toolbox publish agent-tools <version>
```

One new version regardless of how many connections you attach in one call. `create` publishes it as the first (default) version; `connection add` leaves the default unchanged until you `publish`.

## Tools the CLI does NOT manage today

`azd ai toolbox` only handles connection-backed tools (`RemoteTool`, `CognitiveSearch`, `RemoteA2A`, `GroundingWithCustomSearch`). These built-ins have no connection and are NOT addable via this CLI: `web_search`, `code_interpreter`, `file_search`, `function`, `toolbox_search_preview`.

To include any built-in in a toolbox today, use the Python / .NET / JS SDK or call the REST API directly.

## Token and RBAC (agent code)

Token scope: `https://ai.azure.com/.default`. RBAC: the calling identity (developer + agent identity at runtime) needs **Foundry User** on the Foundry project.

## Agent code (Python, Microsoft Agent Framework)

```python
import os, httpx
from azure.identity import DefaultAzureCredential
from agent_framework.tools.mcp import MCPStreamableHTTPTool

_credential = DefaultAzureCredential()

def _inject_auth(request: httpx.Request) -> None:
    # Per-request token refresh -- static tokens expire in ~1 hour.
    token = _credential.get_token("https://ai.azure.com/.default").token
    request.headers["Authorization"] = f"Bearer {token}"

tool = MCPStreamableHTTPTool(
    name="github",                    # becomes server_label prefix
    url=os.environ["TOOLBOX_AGENT_TOOLS_MCP_ENDPOINT"],
    httpx_client=httpx.AsyncClient(event_hooks={"request": [_inject_auth]}),
    load_prompts=False,               # Foundry doesn't implement prompts/list
    approval_mode="never_require",    # for require_approval:always tools
)
```

Install: `pip install httpx azure-identity agent-framework`.

## MCP client gotchas

- **Always stream.** Non-streaming is not supported.
- **Don't call `prompts/list`.** Returns `500`. Pass `load_prompts=False`.
- **Don't `send_ping()`** with generic clients (returns `500`). Agent Framework handles this.
- **Tool names are prefixed with `server_label`.** `name="myserver"` -> tools appear as `myserver___<tool>` (joined by three underscores).
- **`require_approval`** is the client's responsibility -- the toolbox proxy does NOT enforce it. Pass `approval_mode="never_require"` or wire an approval handler.

## Verify the wire end-to-end

```bash
azd ai toolbox list --output json
azd ai toolbox show agent-tools --output json
azd deploy
azd ai agent invoke "list the tools you have access to"
```

## Troubleshooting

| Symptom | Likely cause |
|---------|--------------|
| `TOOLBOX_<NAME>_MCP_ENDPOINT` not set | Run `azd ai toolbox show` + `azd env set`. |
| Env var missing in deployed agent | Add to the agent service's `environmentVariables` in `azure.yaml`, `azd deploy`. |
| `401` on MCP calls | Expired / wrong-scope token. Use `https://ai.azure.com/.default`; refresh per request. |
| `403 Forbidden` | Caller missing `Foundry User` role. |
| `500` on `prompts/list` / ping | Disable in MCP client (`load_prompts=False`). |
| Empty response, tool never called | `require_approval: always` with no handler. Pass `approval_mode="never_require"`. |
| `tools/list` returns zero | Bad credentials, or toolbox version still provisioning. |
| Tool names don't match | Use `{server_label}___{tool_name}` (three underscores). |

# Use Toolbox in a Hosted Agent

Hosted agents access Foundry-managed tools through a **Toolbox MCP endpoint**. Unlike prompt agents that wire tools directly, hosted agents connect to a single MCP-compatible endpoint that exposes all configured tools. The platform handles credential injection, token refresh, and policy enforcement.

> 🚦 **Toolbox creation gate:** before creating a toolbox/connection, you MUST read the boundary rules in [create-hosted.md → Toolbox creation boundary](../create-hosted.md#toolbox-creation-boundary) and follow them, then continue with the rest of this file.

> 📘 For endpoint format, MCP protocol details, auth, OAuth consent handling, testing, citation pattern, and troubleshooting, see [toolbox-reference.md](toolbox-reference.md).
>
> 📘 For wiring a remote tool (catalog tile or generic MCP server) into a project connection that a toolbox can attach to, see [foundry-tool-catalog.md](foundry-tool-catalog.md).
>
> 📘 For the full list of supported tool types and their per-type fields, see [agent-tools.md](agent-tools.md) and the per-tool `tool-*.md` files.

> 💡 **This skill is scoped to *consuming* an existing toolbox from agent code** — endpoint resolution, env-var contract, payload shape gathered before agent runtime, verification, and tracing. **Toolbox and connection CRUD belongs in [Foundry Toolkit (VS Code)](https://code.visualstudio.com/docs/intelligentapps/tool-catalog) or the [Foundry Portal](https://ai.azure.com/)** — those surfaces give you tool browsing, metadata, connection wizards, and validation. Use the imperative `azd ai` CLI only for *operational* tasks (retarget the default version, smoke-test an endpoint).

## ✨ Recommendation: enable Tool Search

**Before adding more than ~5 tools to a toolbox, add `{ "type": "toolbox_search_preview" }` to the toolbox.** This replaces the full `tools/list` shown to the model with two meta-tools — `tool_search` (natural-language discovery) and `call_tool` (invoke a discovered tool) — so context cost stays flat as the toolbox grows.

- The `toolbox_search_preview` entry **doesn't count** toward the unnamed-tool-per-type limit.
- All other tools in the toolbox are hidden from the initial `tools/list` and surfaced only by `tool_search` (or by per-user auto-pinning of hot tools).
- Pin specific high-traffic tools or add ranking-only keywords via `tool_configs.{tool_name}` (with `pin: true` and `additional_search_text`).
- In the agent's system prompt, instruct the model to call `tool_search` whenever a needed capability isn't already visible.

Full configuration recipe in [tool-tool-search.md](tool-tool-search.md) and the public [Tool Search (preview) docs](https://learn.microsoft.com/azure/foundry/agents/how-to/tools/tool-search).

## Quick Reference

| Property | Value |
|----------|-------|
| **Toolbox Docs** | https://learn.microsoft.com/azure/foundry/agents/how-to/tools/toolbox |
| **Tool Catalog Docs** | https://learn.microsoft.com/azure/foundry/agents/concepts/tool-catalog |
| **Tool Search Docs** | https://learn.microsoft.com/azure/foundry/agents/how-to/tools/tool-search |
| **Foundry Toolkit (VS Code) — set up tools/toolboxes** | https://code.visualstudio.com/docs/intelligentapps/tool-catalog |
| **Foundry Portal** | https://ai.azure.com/ |
| **Default Sample (Python, Agent Framework + toolbox)** | https://github.com/microsoft-foundry/foundry-samples/tree/main/samples/python/hosted-agents/agent-framework/responses/04-foundry-toolbox |
| **Python Hosted Agent — `responses` (BYO)** | https://github.com/microsoft-foundry/foundry-samples/tree/main/samples/python/hosted-agents/bring-your-own/responses |
| **Python Hosted Agent — `invocations` (BYO)** | https://github.com/microsoft-foundry/foundry-samples/tree/main/samples/python/hosted-agents/bring-your-own/invocations |
| **C# (.NET) Hosted Agent + toolbox** | https://github.com/microsoft-foundry/foundry-samples/tree/main/samples/csharp/hosted-agents/agent-framework/foundry-toolbox-server-side |
| **Supported Toolbox Scenarios (sample-side reference)** | https://github.com/microsoft-foundry/foundry-samples/blob/main/samples/python/hosted-agents/SUPPORTED_TOOLBOX_SCENARIOS.md |

## Resolve Toolbox Endpoint

If the user provides a toolbox name or endpoint URL, or the project already references a toolbox (e.g., in `.env` or `agent.manifest.yaml`) → use it directly.

Otherwise, ask one question:

> _"Would you like to provide your toolbox endpoint? (you can create one with the [Foundry Toolkit in VS Code](https://code.visualstudio.com/docs/intelligentapps/tool-catalog) or the [Foundry Portal](https://ai.azure.com/))"_

Once the user supplies the toolbox name/endpoint — either an existing one or a new one they create via the Foundry Toolkit or Foundry Portal — set it on the agent (e.g., `TOOLBOX_ENDPOINT` in `.env`) and continue with verification.

> Use the env var name **`TOOLBOX_ENDPOINT`** (no `FOUNDRY_` prefix). The Foundry platform reserves `FOUNDRY_`-prefixed env vars and may silently overwrite them at runtime — see [toolbox-reference.md § Agent env contract](toolbox-reference.md#agent-env-contract).

> **When asking the question, always include the doc links inline** for the manual options — the [Foundry Toolkit in VS Code](https://code.visualstudio.com/docs/intelligentapps/tool-catalog) and the [Foundry Portal](https://ai.azure.com/) — so the user knows where to go to create a tool/toolbox themselves. Don't just name the options; render them as clickable links every time.

> **Before printing out any step-by-step guidance** for the Foundry Toolkit (VS Code) path, fetch and read [Use Tool Catalog to connect tools and Toolboxes in Foundry Toolkit](https://code.visualstudio.com/docs/intelligentapps/tool-catalog) first, then summarize the relevant steps for them. Don't paraphrase from memory — the Toolkit UI changes; quote the current doc.

## Available tool types

The full set is documented in [agent-tools.md](agent-tools.md) and — authoritatively — in the public [Toolbox docs (Configure tools)](https://learn.microsoft.com/azure/foundry/agents/how-to/tools/toolbox#configure-tools). At time of writing the supported `type` values are:

| `type` | Tool | Connection required? | Detail |
|---|---|---|---|
| `mcp` | Remote MCP server (third-party via catalog, BYO OAuth, or generic) | Optional (none / static key / project MI / OAuth) | [tool-mcp.md](tool-mcp.md) |
| `web_search` | Web search (basic Bing; optional `web_search.custom_search_configuration` for Bing Custom Search to scope grounding to specific domains) | No (basic); Yes for Custom Search | [tool-web-search.md](tool-web-search.md) |
| `azure_ai_search` | Azure AI Search index | Yes (Search service connection) | [tool-azure-ai-search.md](tool-azure-ai-search.md) |
| `code_interpreter` | Sandboxed Python execution | No | [tool-code-interpreter.md](tool-code-interpreter.md) |
| `file_search` | Vector-store-backed retrieval over uploaded files | No (vector store is part of the toolbox) | [tool-file-search.md](tool-file-search.md) |
| `openapi` | REST API exposed via an OpenAPI 3.x spec | Conditional (`connection` requires `project_connection_id`; `managed_identity` does not — uses project MI + `audience`) | [tool-openapi.md](tool-openapi.md) |
| `a2a_preview` | Call another Foundry agent as a tool | Optional | [tool-a2a.md](tool-a2a.md) |
| `work_iq_preview` | Microsoft 365 work context (mail / meetings / files / chats) via Work IQ | Yes (Work IQ `RemoteA2A` OAuth connection; BYO Entra app; M365 Copilot license per user) | [tool-work-iq.md](tool-work-iq.md) |
| `fabric_iq_preview` | Microsoft Fabric data (Ontology / Fabric data agent / Power BI semantic model) | Yes (Fabric IQ OAuth connection; tenant admin consent) | [tool-fabric-iq.md](tool-fabric-iq.md) |
| `toolbox_search_preview` | **Tool Search** — a directive (not a tool) that swaps `tools/list` for `tool_search` + `call_tool` meta-tools | No | [tool-tool-search.md](tool-tool-search.md) |

**Adjacent (not a `type` in a toolbox version):**

- **Agent Memory** — use the `MemorySearchTool` SDK class on prompt agents; for hosted agents, configure the memory store via the project (separate from the toolbox). See [tool-memory.md](tool-memory.md).
- **Routines (preview)** — not a tool; an agent **trigger** (`schedule` / `timer` / `github_issue` / `custom`) that invokes an existing agent. See the [public Routines docs](https://learn.microsoft.com/azure/foundry/agents/how-to/use-routines).

## Information to Gather Before Building a Toolbox Payload

When the user asks to "add an MCP tool" or similar, **never guess**. Confirm each field before generating any JSON or `azure.yaml` snippet:

| # | Question | Why needed |
|---|----------|------------|
| 1 | **MCP server URL?** | The `server_url` field on the `mcp` tool entry |
| 2 | **Auth type?** `none` / `key` / `mi` / `oauth` | Determines whether a project connection is required and which shape to create (see [foundry-tool-catalog.md](foundry-tool-catalog.md)) |
| 3 | **Project connection name** (if auth ≠ `none`) | The `project_connection_id` field; must already exist in the Foundry project |
| 4 | **`server_label`** | Short prefix for the tool names exposed by this server (e.g. `myserver`) |
| 5 | **Toolbox name** | The container that will hold the tool entries |
| 6 | **Foundry project endpoint** | Where the toolbox is created — read from `PROJECT_ENDPOINT` / `AZURE_AI_PROJECT_ENDPOINT` (avoid `FOUNDRY_`-prefixed names) |
| 7 | **Many tools planned?** (> ~5) | If yes, also add `{ "type": "toolbox_search_preview" }` so the model uses [Tool Search](#-recommendation-enable-tool-search) instead of seeing the full list. |

### Toolbox payload — MCP with a project connection

```json
{
  "name": "<TOOLBOX_NAME>",
  "description": "MCP server with key or OAuth auth",
  "tools": [
    {
      "type": "mcp",
      "server_label": "<LABEL>",
      "server_url": "<SERVER_URL>",
      "require_approval": "never",
      "project_connection_id": "<CONNECTION_NAME>"
    }
  ]
}
```

### Toolbox payload — public MCP (no auth)

```json
{
  "name": "api-specs",
  "description": "Public MCP server, no connection needed",
  "tools": [
    {
      "type": "mcp",
      "server_label": "api_specs",
      "server_url": "https://gitmcp.io/Azure/azure-rest-api-specs",
      "require_approval": "never"
    }
  ]
}
```

### Toolbox payload — large toolbox with Tool Search

```json
{
  "name": "big-toolbox",
  "description": "Many tools — model uses tool_search to discover",
  "tools": [
    { "type": "toolbox_search_preview" },
    { "type": "web_search" },
    { "type": "azure_ai_search", "name": "docs_index", "project_connection_id": "search-conn", "index_name": "docs" },
    {
      "type": "mcp", "server_label": "github", "server_url": "<github-mcp-url>",
      "project_connection_id": "gh-conn",
      "tool_configs": {
        "search_issues": { "pin": true, "additional_search_text": "GitHub issues bug tracking" },
        "*":             { "additional_search_text": "GitHub repositories code" }
      }
    }
  ]
}
```

### Declarative path via `azd`

If the project already uses `azd ai agent init`, prefer declaring the toolbox in `azure.yaml` so `azd deploy` provisions it and injects `TOOLBOX_ENDPOINT` automatically:

```yaml
# Declare secret parameters first; azd will prompt for the value on `azd up`
# (or read it from `AZURE_<NAME>` env vars) and never store it in plaintext.
params:
  - name: github_pat
    type: securestring

resources:
  - kind: connection
    name: <CONNECTION_NAME>
    target: <MCP_SERVER_URL>
    category: remoteTool
    credentials:
      type: CustomKeys
      keys:
        # Header name comes from the catalog entry's x-ms-connection-parameters.
        # {{ github_pat }} is resolved from the `params` block above.
        Authorization: "Bearer {{ github_pat }}"

  - kind: toolbox
    name: agent-tools
    tools:
      - type: toolbox_search_preview   # recommended for any toolbox > ~5 tools
      - type: web_search
      - type: mcp
        server_label: <LABEL>
        server_url: <MCP_SERVER_URL>
        project_connection_id: <CONNECTION_NAME>
```

See [azd `params` reference](https://learn.microsoft.com/azure/developer/azure-developer-cli/azd-schema#params) for the full parameter syntax.

## Operational helpers via `azd ai` CLI

> The `azd ai` CLI also exposes `connection create`, `toolbox create`, `toolbox list`, and `toolbox delete`. Prefer **Foundry Toolkit (VS Code)** or the **Foundry Portal** for those — the UI gives you tool browsing, connection wizards, and validation. The two commands below are the ones the skill should still drive directly because they're *operational*, not setup.

> All commands require `--project-endpoint <PROJECT_ENDPOINT>` (the value of `PROJECT_ENDPOINT`, e.g. `https://<account>.services.ai.azure.com/api/projects/<project>`). To avoid repeating it, export it once:
>
> ```pwsh
> $PE = "https://<account>.services.ai.azure.com/api/projects/<project>"
> ```

### Retarget the default version — `azd ai toolbox publish`

Each toolbox version is **immutable**. The version an agent actually hits is the one marked `*` in `versions list` — i.e. the **default version**. Use `publish` to point that pointer at any existing version (e.g. rollback to a known-good version after a bad publish).

```pwsh
# Inspect first — current default is marked with '*'
azd ai toolbox versions list my-toolbox --project-endpoint $PE

# Retarget the default
azd ai toolbox publish my-toolbox 20 --project-endpoint $PE --no-prompt

# Verify (Default version / Shown version / Endpoint all reflect the new value)
azd ai toolbox show my-toolbox --project-endpoint $PE
```

- `publish <name> <version>` promotes an existing version to default (also used to roll back).
- Validated: switched `default-tb` from version 21 → 20 → 21; both `show` and the computed MCP endpoint (`.../toolboxes/<name>/versions/<n>/mcp?api-version=v1`) tracked the change immediately.

### End-to-end smoke test

After the toolbox is created (via Toolkit / Portal / `azd`), hit the MCP endpoint directly to confirm the tool is reachable before pointing an agent at it:

```pwsh
$TOK = az account get-access-token --resource "https://ai.azure.com" --query accessToken -o tsv
$H   = @{
  Authorization      = "Bearer $TOK"
  "Content-Type"     = "application/json"
}
$URL = "$PE/toolboxes/my-toolbox/mcp?api-version=v1"
$body = @{ jsonrpc = "2.0"; id = 1; method = "tools/list"; params = @{} } | ConvertTo-Json
(Invoke-RestMethod -Method POST -Uri $URL -Headers $H -Body $body).result.tools | Select-Object name
```

`?api-version=v1` is required.

## Code Integration Patterns

The sample repo provides integration patterns for both Python and C#. Read the sample code and adapt it to the user's project.

**Python samples:**

| Sample | Framework | Protocol | When to use |
|--------|-----------|----------|-------------|
| [`agent-framework/responses/04-foundry-toolbox/`](https://github.com/microsoft-foundry/foundry-samples/tree/main/samples/python/hosted-agents/agent-framework/responses/04-foundry-toolbox) — recommended | Agent Framework (MAF) | Responses | **Default choice** |
| [`bring-your-own/responses/langgraph-toolbox/`](https://github.com/microsoft-foundry/foundry-samples/tree/main/samples/python/hosted-agents/bring-your-own/responses/langgraph-toolbox) | LangGraph (BYO) | Responses | LangGraph hosted agent with toolbox |
| [`bring-your-own/responses/bring-your-own-toolbox/`](https://github.com/microsoft-foundry/foundry-samples/tree/main/samples/python/hosted-agents/bring-your-own/responses/bring-your-own-toolbox) | Generic MCP (BYO) | Responses | Raw `httpx` MCP client — works with any framework |
| [`bring-your-own/invocations/toolbox/`](https://github.com/microsoft-foundry/foundry-samples/tree/main/samples/python/hosted-agents/bring-your-own/invocations/toolbox) | Generic MCP (BYO) | Invocations | Toolbox via Invocations protocol |

**C# (.NET) samples:**

| Sample | Description |
|--------|-------------|
| [`csharp/hosted-agents/agent-framework/foundry-toolbox-server-side/`](https://github.com/microsoft-foundry/foundry-samples/tree/main/samples/csharp/hosted-agents/agent-framework/foundry-toolbox-server-side) — recommended | Agent Framework agent with toolbox MCP (Responses protocol) |

**Notes** (apply to all patterns, both Python and C#):

- Auth: Inject a bearer token with scope `https://ai.azure.com/.default` on every request (Python: `httpx.Auth` subclass; C#: `DefaultAzureCredential` + `BearerTokenAuthenticationPolicy`).
- MCP client: Pass `load_prompts=False` — the toolbox endpoint does not support `prompts/list`.
- Endpoint: Construct from `{project_endpoint}/toolboxes/{toolbox_name}/mcp?api-version=v1`.
- Multi-tool toolboxes: at most one tool per unnamed type, and unique `server_label` per MCP tool (see [toolbox-reference.md](toolbox-reference.md#multi-tool-toolbox-constraint)). `toolbox_search_preview` doesn't count toward this limit.
- Tool naming: MCP-sourced tools are prefixed `{server_label}___{tool_name}` (three underscores); **all other tool types** use the entry's `name` field value (or the default tool name).

> 💡 **Tip:** If MCP tools have `require_approval: "always"` in `_meta.tool_configuration`, the agent runtime must ask the user for confirmation before invoking. The toolbox endpoint does not enforce this — your agent code is responsible.

## Tracing

All toolbox samples emit OpenTelemetry traces. No code changes are required to enable export to Azure Monitor — it's purely a configuration step.

- **Local development:** set `APPLICATIONINSIGHTS_CONNECTION_STRING` in the agent's `.env`.
- **Deployed:** the platform injects `APPLICATIONINSIGHTS_CONNECTION_STRING` automatically when the Foundry project is linked to an Application Insights resource.
- **Per-framework instrumentation hooks** (already present in the samples):
  - `maf` — `main.py` calls `enable_instrumentation()`.
  - `langgraph` / `azd` — auto-instrumented by `azure-ai-agentserver-core[tracing]`.
- **Viewing traces:** Azure Portal → Application Insights → **Investigate → Transaction search** (per-trace) or **Application map** (dependency graph).

# Create a Guardrail via the REST API (`az rest`)

> Use this path only when the user explicitly asks for programmatic/CLI/CI/CD creation. Otherwise, guide them to the [portal](guardrail-manage.md#default-path-portal).

## Prerequisites

- Azure CLI installed and logged in (`az login`)
- **Foundry Account Owner** role (or higher) on the Azure AI resource
- The Azure AI Services account name, resource group, and subscription ID

## Step 1: Set Variables

```bash
SUBSCRIPTION_ID=$(az account show --query id -o tsv)
RESOURCE_GROUP="<your-resource-group>"
ACCOUNT_NAME="<your-ai-services-account>"
POLICY_NAME="my-custom-guardrail"
```

## Step 2: List Existing Guardrails

```bash
az rest --method GET \
  --url "https://management.azure.com/subscriptions/${SUBSCRIPTION_ID}/resourceGroups/${RESOURCE_GROUP}/providers/Microsoft.CognitiveServices/accounts/${ACCOUNT_NAME}/raiPolicies?api-version=2024-10-01"
```


## Step 3: Create a Guardrail

Minimal example — `guardrail-policy.json`:

```json
{
  "properties": {
    "basePolicyName": "Microsoft.Default",
    "mode": "Asynchronous_filter",
    "contentFilters": [
      { "name": "Hate", "enabled": true, "blocking": true, "severityThreshold": "Medium", "source": "Prompt" },
      { "name": "Hate", "enabled": true, "blocking": true, "severityThreshold": "Medium", "source": "Completion" },
      { "name": "Violence", "enabled": true, "blocking": true, "severityThreshold": "Low", "source": "Prompt" },
      { "name": "Violence", "enabled": true, "blocking": true, "severityThreshold": "Low", "source": "Completion" },
      { "name": "Jailbreak", "enabled": true, "blocking": true, "source": "Prompt" }
    ]
  }
}
```

```bash
az rest --method PUT \
  --url "https://management.azure.com/subscriptions/${SUBSCRIPTION_ID}/resourceGroups/${RESOURCE_GROUP}/providers/Microsoft.CognitiveServices/accounts/${ACCOUNT_NAME}/raiPolicies/${POLICY_NAME}?api-version=2024-10-01" \
  --body @guardrail-policy.json
```

- `200 OK` — policy updated
- `201 Created` — policy created for the first time

For the full request body schema (`contentFilters[]`, `basePolicyName`, `mode`, `customBlocklists`), see the [RAI Policies - Create Or Update API reference](https://learn.microsoft.com/rest/api/aiservices/accountmanagement/rai-policies/create-or-update).

## Step 4: Verify via CLI

```bash
az rest --method GET \
  --url "https://management.azure.com/subscriptions/${SUBSCRIPTION_ID}/resourceGroups/${RESOURCE_GROUP}/providers/Microsoft.CognitiveServices/accounts/${ACCOUNT_NAME}/raiPolicies/${POLICY_NAME}?api-version=2024-10-01"
```

Confirm `contentFilters[]` matches your intended configuration.

## Step 5: Assign to Targets

After creating a guardrail, assign it to a hosted agent, model deployment, or toolbox → [guardrail-attach.md](guardrail-attach.md)

## Delete a Guardrail

> Remove all model/agent assignments before deleting (portal: edit guardrail → deselect all targets → save).

```bash
az rest --method DELETE \
  --url "https://management.azure.com/subscriptions/${SUBSCRIPTION_ID}/resourceGroups/${RESOURCE_GROUP}/providers/Microsoft.CognitiveServices/accounts/${ACCOUNT_NAME}/raiPolicies/${POLICY_NAME}?api-version=2024-10-01"
```

## References

- [RAI Policies - Create Or Update (REST API)](https://learn.microsoft.com/rest/api/aiservices/accountmanagement/rai-policies/create-or-update) — full schema, parameters, response codes
- [RAI Policies - List](https://learn.microsoft.com/rest/api/aiservices/accountmanagement/rai-policies/list) — list all policies on an account
- [Blocklists API](https://learn.microsoft.com/rest/api/aiservices/accountmanagement/rai-blocklists) — custom blocklists (created separately)
- [How to configure guardrails (Microsoft Learn)](https://learn.microsoft.com/azure/foundry/guardrails/how-to-create-guardrails) — portal walkthrough

# Attach a Guardrail

After creating a guardrail (via [portal](guardrail-manage.md) or [REST API](guardrail-api-create.md)), attach it to one of three targets:

- [Hosted Agent](#hosted-agent) — `agent.yaml` `policies` block
- [Model Deployment](#model-deployment) — REST API or request-time header
- [Toolbox](#toolbox) — `policies.rai_config.rai_policy_name` in toolbox definition

---

## Hosted Agent

A guardrail assigned to an agent **fully overrides** the underlying model deployment's guardrail. If no guardrail is assigned, the agent inherits the model deployment's guardrail.

Add a `policies` block to `agent.yaml` with the guardrail's full ARM resource ID:

```yaml
policies:
  - type: rai_policy
    rai_policy_name: /subscriptions/<sub-id>/resourceGroups/<rg>/providers/Microsoft.CognitiveServices/accounts/<account>/raiPolicies/<policy-name>
```

See the [`16-content-safety-guardrail` sample](https://github.com/microsoft-foundry/foundry-samples/tree/main/samples/python/hosted-agents/agent-framework/responses/16-content-safety-guardrail) for a complete working example.

> `rai_policy_name` must be the **full ARM resource ID**, not just the policy name. This differs from the toolbox and model deployment paths which use just the name.

---

## Model Deployment

### Assign via REST API

```bash
SUBSCRIPTION_ID=$(az account show --query id -o tsv)
RESOURCE_GROUP="<your-resource-group>"
ACCOUNT_NAME="<your-ai-services-account>"
DEPLOYMENT_NAME="<your-model-deployment>"

az rest --method PATCH \
  --url "https://management.azure.com/subscriptions/${SUBSCRIPTION_ID}/resourceGroups/${RESOURCE_GROUP}/providers/Microsoft.CognitiveServices/accounts/${ACCOUNT_NAME}/deployments/${DEPLOYMENT_NAME}?api-version=2024-10-01" \
  --body '{"properties": {"raiPolicyName": "my-custom-guardrail"}}'
```

> `raiPolicyName` is the guardrail name (not the full ARM resource ID). It must match a guardrail that exists on the AI Services account.

### Request-Time Override

Override the deployment-level guardrail per request using the `x-policy-id` header:

```bash
ENDPOINT="https://<your-resource-name>.openai.azure.com"
DEPLOYMENT_NAME="<your-model-deployment>"
API_KEY="<your-api-key>"

curl --request POST \
  --url "${ENDPOINT}/openai/deployments/${DEPLOYMENT_NAME}/chat/completions?api-version=2024-10-21" \
  --header "Content-Type: application/json" \
  --header "api-key: ${API_KEY}" \
  --header "x-policy-id: my-custom-guardrail" \
  --data '{
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "Hello!"}
    ]
  }'
```

> Request-time override is not available for image input scenarios.

---

## Toolbox

Add `policies.rai_config.rai_policy_name` to the toolbox definition file, then create the toolbox with `azd ai toolbox create`.

```yaml
description: My toolbox
connections:
  - name: my-mcp-server
tools:
  - type: web_search
    name: web
policies:
  rai_config:
    rai_policy_name: my-custom-guardrail
```

> `rai_policy_name` must match a guardrail that exists on the AI Services account. Use `Microsoft.Default`, `Microsoft.DefaultV2`, or a custom name created via [portal or API](guardrail-api-create.md).

```bash
azd ai toolbox create my-toolbox --from-file ./toolbox.yaml
```

There is no command to change the guardrail on an existing toolbox version. To update, delete and recreate the toolbox.

---

## References

- [Guardrails overview](guardrail-manage.md) — create guardrails, default policies, intervention points
- [API create](guardrail-api-create.md) — create guardrails via REST API
- [Guardrails overview (Microsoft Learn)](https://learn.microsoft.com/azure/foundry/guardrails/guardrails-overview)
- [How to configure guardrails (Microsoft Learn)](https://learn.microsoft.com/azure/foundry/guardrails/how-to-create-guardrails)

# Guardrails (RAI Content-Filter Policies)

Guardrails are Responsible AI (RAI) content-filter policies that control what content is allowed through model deployments and agents in Microsoft Foundry.

## When to Use

- Create or manage a guardrail (content-filter policy) for a Foundry project
- Attach a guardrail to a hosted agent, model deployment, or toolbox → [guardrail-attach.md](guardrail-attach.md)
- Create a guardrail via REST API → [guardrail-api-create.md](guardrail-api-create.md)

## Default Path: Portal

By default, guide the user to the Foundry portal to create guardrails interactively.

**Construct and show this URL to the user:**

```
https://ai.azure.com/nextgen/r/{encodedSubId},{resourceGroup},,{accountName},{projectName}/build/guardrails
```

Where:
- `{encodedSubId}` — subscription GUID as URL-safe base64 (no `=` padding):
  ```bash
  python -c "import base64,uuid;print(base64.urlsafe_b64encode(uuid.UUID('<SUBSCRIPTION_ID>').bytes).rstrip(b'=').decode())"
  ```
- `{resourceGroup}` — resource group name
- `{accountName}` — AI Services account name
- `{projectName}` — Foundry project name

If resource details are unknown, use the generic URL and instruct the user to navigate manually:

```
https://ai.azure.com
```

Then navigate: select your project → **Build** → **Guardrails** → **Create Guardrail**.

> Use the API path ([guardrail-api-create.md](guardrail-api-create.md)) only when the user explicitly asks for programmatic/CLI/CI/CD creation.

## Intervention Points

| Intervention Point | Models | Agents (Preview) | Toolbox |
|---|---|---|---|
| User input | Yes | Yes | No |
| Tool call | No | Yes | Yes |
| Tool response | No | Yes | Yes |
| Output | Yes | Yes | No |

Tool call and tool response are agent-only (and toolbox). An agent's guardrail fully overrides its model deployment's guardrail at all intervention points.

## Default Guardrails

| Policy Name | Description | Editable |
|-------------|-------------|----------|
| `Microsoft.Default` | Base default policy (4 categories) | No |
| `Microsoft.DefaultV2` | Updated default with jailbreak + protected material | No |
| Custom policies | User-created policies | Yes |

# Use Skills in a Hosted Agent

How to consume Foundry **skills** (reusable behavioral guidelines) from hosted agent code. Two approaches:

1. **Direct download** — agent downloads skill ZIPs at startup via the Skills API and builds a skills provider.
2. **Via Toolbox MCP** — agent connects to a toolbox MCP endpoint that exposes skills as resources.

## How progressive disclosure works

The Agent Framework SDK injects skill names/descriptions into the system prompt (~100 tokens each) and synthesizes a `load_skill` tool. When the model determines a skill is relevant, it calls `load_skill(name)` to retrieve the full body on demand — keeping context usage low.

## Choosing an approach

| | Direct Download | Via Toolbox MCP |
|--|---|---|
| How | Downloads ZIPs at startup, builds provider from local files | Connects to toolbox MCP; SDK reads `resources/list` → `load_skill` |
| Skill updates | Redeploy agent | Consumer endpoint picks up new version automatically |
| Header | `Foundry-Features: Skills=V1Preview` | Not required |
| When to use | No toolbox; need explicit version control | Already have a toolbox; want dynamic updates |

---

## Approach 1: Direct Download

At startup, the agent downloads skill ZIPs from the Foundry Skills API, extracts them to a local directory, and builds a skills provider. The SDK advertises skill names/descriptions in the system prompt and synthesizes a `load_skill` tool for on-demand loading.

**Prerequisites:**
- Skills provisioned in the Foundry project — see [skill-manage.md](skill-manage.md)

**Env vars** — set in `.env` for local run, and in the agent service's `environmentVariables` in `azure.yaml` for deployed agents:

| Variable | Purpose |
|----------|---------|
| `FOUNDRY_PROJECT_ENDPOINT` | Project endpoint for SDK calls |
| `AZURE_AI_MODEL_DEPLOYMENT_NAME` | Model deployment for the agent |
| `SKILL_NAMES` | Comma-separated skill names to download |

### Python

Flow: download skills → extract ZIPs → build skills provider → attach to agent as context provider → SDK synthesizes `load_skill` tool.

Full working sample: [12-foundry-skills (Python)](https://github.com/microsoft-foundry/foundry-samples/tree/main/samples/python/hosted-agents/agent-framework/responses/12-foundry-skills) — **read `README.md` and `main.py`** for setup and integration details.

### C#

Flow: download skills via Skills API → extract ZIPs → build skills provider → register in agent context → SDK synthesizes `load_skill` tool.

Full working sample: [agent-skills (C#)](https://github.com/microsoft-foundry/foundry-samples/tree/main/samples/csharp/hosted-agents/agent-framework/agent-skills) — **read `README.md` and `Program.cs`** for setup and integration details.

---

## Approach 2: Via Toolbox MCP

The agent connects to a toolbox MCP endpoint at startup. The SDK discovers skills via `resources/list`, advertises them in the system prompt, and synthesizes a `load_skill` tool that reads skill content via `resources/read` on demand.

**Prerequisites:**
- Skills provisioned in the Foundry project — see [skill-manage.md](skill-manage.md)
- Skills attached to a toolbox — see [skill-toolbox-attach.md](skill-toolbox-attach.md)

**Env vars** — set in `.env` for local run, and in the agent service's `environmentVariables` in `azure.yaml` for deployed agents:

| Variable | Purpose |
|----------|---------|
| `FOUNDRY_PROJECT_ENDPOINT` | Project endpoint for SDK calls |
| `AZURE_AI_MODEL_DEPLOYMENT_NAME` | Model deployment for the agent |
| `TOOLBOX_NAME` | Toolbox name — SDK constructs endpoint |

### C#

Flow: connect to toolbox MCP endpoint → discover skills via `resources/list` → build skills provider from MCP resources → SDK synthesizes `load_skill` tool → reads skill content via `resources/read` on demand.

Full working sample: [foundry-toolbox-mcp-skills (C#)](https://github.com/microsoft-foundry/foundry-samples/tree/main/samples/csharp/hosted-agents/agent-framework/foundry-toolbox-mcp-skills) — **read `README.md` and `Program.cs`** for setup and integration details.

## Verify end-to-end

```bash
azd ai agent run
azd ai agent invoke --local "Hi, can I return my tent within 30 days?"
```

### Handling `mcp_approval_request`

The `load_skill` tool is exposed as an MCP tool in the Responses protocol. The sample code defaults to requiring approval (`require_approval: "always"`), so the agent returns both a `function_call` (completed) and an `mcp_approval_request` in the output. The agent will not produce a text response until the client approves the request.

**Foundry Portal** — after deploying, the portal playground shows the approval prompt and handles it interactively.

**Local with Agent Inspector** (`azd ai agent run`) — the Inspector UI shows an approval button to approve the request.

**Local without Inspector** (`azd ai agent run --no-inspector`) — use `curl` against `http://localhost:8088/responses` directly:

1. Send the initial message:

```bash
curl -s -X POST http://localhost:8088/responses \
  -H "Content-Type: application/json" \
  -d '{"input": "What is your return policy?"}'
```

The response includes `mcp_approval_request` items with an `id` field.

2. Approve and continue — send `mcp_approval_response` referencing each `id`:

```bash
curl -s -X POST http://localhost:8088/responses \
  -H "Content-Type: application/json" \
  -d '{
    "previous_response_id": "<response_id_from_step_1>",
    "input": [
      {
        "type": "mcp_approval_response",
        "approval_request_id": "<mcp_approval_request_id>",
        "approve": true
      }
    ]
  }'
```

The agent now produces the text response with skills applied.

**To skip approval entirely**, configure `require_approval: "never"` on the MCP tool. This behavior is controlled by the Agent Framework SDK — see the sample code for how MCP tool approval is configured.

## Troubleshooting

| Symptom | Likely cause | Fix |
|---------|------------|-----|
| `SKILL.md not found` after download | ZIP doesn't contain `SKILL.md` at root | Create skill from directory with `SKILL.md` at root |
| Agent ignores skills | Descriptions don't match user queries | Improve `description` in SKILL.md front matter |
| Skills load but agent doesn't follow | Instructions vague or conflicting | Refine skill body; add canary token to verify loading |
| `asyncio.TimeoutError` (Python) | Slow network or large packages | Increase bootstrap timeout (default 60s) |
| `allow_preview` error (Python) | SDK client missing preview flag | Set `allow_preview=True` on the project client |
| HTTP 500 on skill download (C#) | Missing feature header | Add `Skills=V1Preview` feature header to requests |
| `SKILL_NAMES` not in deployed agent | Env var missing from `azure.yaml` | Add to the agent service's `environmentVariables`, redeploy |
| MCP timeout (Toolbox) | Auth token expired or wrong scope | Use `https://ai.azure.com/.default`; refresh per request |

## References

- [skill-manage.md](skill-manage.md) — create, version, and manage skills
- [skill-toolbox-attach.md](skill-toolbox-attach.md) — attach skills to a toolbox, MCP protocol

# Skills (azd ai)

How to create, manage, and version **skills** (reusable behavioral guidelines) in a Foundry project using `azd ai skill` CLI and SDK.

A **skill** is a Markdown file with YAML front matter (`SKILL.md`), uploaded to a Foundry project, and attached to agents at runtime. Skills enable updating agent behavior **without code changes**.

## Install the extension

```bash
azd extension install azure.ai.skills
```

## Skill authoring format

Each skill lives in its own directory with `SKILL.md` at the root:

```
skills/
  my-skill/
    SKILL.md       # YAML front matter + Markdown body
```

```yaml
---
name: my-skill-name
description: What this skill does and when the agent should load it
---

# My Skill

Instructions the agent follows when this skill is loaded on demand...
```

> **The `name` and `description` values must be unquoted** in YAML front matter — quoting causes HTTP 500 on import.

The `description` field drives skill discovery at runtime: the Agent Framework SDK uses it to decide when to load the skill. Write descriptions that clearly state **when** the agent should use the skill. See [skill-attach.md § How progressive disclosure works](skill-attach.md) for details.

## CLI surface — `azd ai skill`

| Command | What it does |
|---------|--------------|
| `azd ai skill create <name> --file <path>` | Create skill + publish v1. Accepts SKILL.md, .zip, or directory. |
| `azd ai skill create <name> --description "..." --instructions "..."` | Inline create (no file). |
| `azd ai skill create <name> --file <path> --force` | Delete existing + recreate. Safe to re-run after edits. |
| `azd ai skill update <name> --file <path>` | New immutable version, promoted to default. |
| `azd ai skill update <name> --set-default-version <ver>` | Repoint default (rollback) without uploading new content. |
| `azd ai skill show <name>` | Show metadata (default_version, latest_version). |
| `azd ai skill list` | List skills in the project. |
| `azd ai skill download <name>` | Extract to `./.agents/skills/<name>/`. |
| `azd ai skill download <name> --version <ver>` | Download a specific version. |
| `azd ai skill download <name> --raw` | Write raw ZIP without extracting. |
| `azd ai skill delete <name> [--force]` | Delete skill. |

Every mutation creates a new immutable version. `create` promotes v1 to default; `update` promotes the new version to default.

Four mutually exclusive input modes for `create` and `update`:

1. **Directory:** `--file ./skills/my-skill/` (CLI packages as ZIP; requires `SKILL.md` at root)
2. **SKILL.md:** `--file ./SKILL.md` (CLI parses YAML front matter + body)
3. **ZIP:** `--file ./skill.zip` (uploaded as multipart/form-data)
4. **Inline:** `--description "..." --instructions "..."` (no file)

## Recipe: create a skill

```bash
azd ai skill create support-style --file ./skills/support-style/
```

## Recipe: batch provision (safe to re-run)

```bash
for dir in skills/*/; do
  name=$(basename "$dir")
  azd ai skill create "$name" --file "$dir" --force
done
```

## Recipe: update a skill

```bash
# Edit SKILL.md locally, then:
azd ai skill update my-skill --file ./skills/my-skill/
```

After update:
- Toolbox skill references (without pinned version) follow the new `default_version` — live immediately, no toolbox republish needed.
- `SkillsProvider` downloads at agent startup — redeploy agent to pick up the new version.

## Recipe: rollback a skill version

```bash
azd ai skill update my-skill --set-default-version 1
```

## Python SDK operations

For programmatic skill CRUD (create, list, download, delete) via the Python SDK, see the provisioning script in the sample:

[provision_skills.py](https://github.com/microsoft-foundry/foundry-samples/blob/main/samples/python/hosted-agents/agent-framework/responses/12-foundry-skills/provision_skills.py) — **read the script source** for the current API surface and usage patterns.

> The Skills SDK API is in preview and may change across versions. Always refer to the sample for the latest usage.

## RBAC

Skills require **Foundry User** on the Foundry project scope (for both the developer identity and the deployed agent's managed identity).

## Versioning

- Every `create` produces version 1 as the default.
- Every `update` creates a new immutable version and promotes it to default.
- `azd ai skill update <name> --set-default-version <ver>` repoints without uploading new content.
- Toolbox skill references without a pinned version follow the skill's `default_version`.
- Toolbox skill references with a pinned version (`skill@2`) stay on that version regardless.
- `SkillsProvider` downloads the `default_version` at agent startup.

## Troubleshooting

| Symptom | Likely cause | Fix |
|---------|------------|-----|
| HTTP 500 on skill create | Quoted `name` or `description` in YAML front matter | Remove quotes from front matter values |
| `403 Forbidden` | Missing RBAC | Grant **Foundry User** on the project scope |
| `azd ai skill` not recognized | Extension not installed | `azd extension install azure.ai.skills` |
| Agent still uses old skill content after `update` | Toolbox skill pinned to old version, or skills provider caches at startup | Use consumer endpoint (no version pin), or redeploy agent |

## References

- [skill-toolbox-attach.md](skill-toolbox-attach.md) — attach skills to a toolbox, MCP protocol
- [skill-attach.md](skill-attach.md) — consume skills in agent code (direct download or toolbox MCP)

# Skills in Toolbox

How to attach, list, remove, and version **skills** (reusable behavioral guidelines) in a Foundry toolbox using `azd ai toolbox skill`.

Skills are not a tool `type` — they live in a separate `skills[]` array in the toolbox manifest. At the MCP level, skills are exposed as **resources** (`resources/list` / `resources/read` with `skill://` URIs).

## Install

```bash
azd extension install azure.ai.skills       # skill CRUD
azd extension install azure.ai.toolboxes    # toolbox management
```

## CLI surface — `azd ai toolbox skill`

| Command | What it does |
|---------|--------------|
| `azd ai toolbox skill add <toolbox> <skill>` | Attach skill (follows default version); new immutable toolbox version. |
| `azd ai toolbox skill add <toolbox> <skill>@<ver>` | Attach skill pinned to a specific version. |
| `azd ai toolbox skill add <toolbox> --from-file <path>` | Attach multiple skills from JSON/YAML. |
| `azd ai toolbox skill list <toolbox>` | List skill references in the toolbox. |
| `azd ai toolbox skill remove <toolbox> <skill> [<skill>...] [--force]` | Detach skills; one new version. |

> Every `skill add` / `skill remove` creates a new immutable toolbox version but does **not** change the default. Run `azd ai toolbox publish <toolbox> <version>` to promote.

## Recipe: attach skill to existing toolbox

```bash
# 1. Create the skill (if not already uploaded)
azd ai skill create support-style --file ./skills/support-style/

# 2. Attach to toolbox
azd ai toolbox skill add agent-tools support-style

# 3. Promote the new toolbox version
azd ai toolbox publish agent-tools <new-version>

# 4. Verify
azd ai toolbox skill list agent-tools
```

## Recipe: include skills in toolbox creation

Skills are a top-level `skills[]` array in the `--from-file` manifest:

```yaml
description: Agent toolbox with skills
connections:
  - name: my-mcp-server
skills:
  - name: support-style
  - name: escalation-policy
    version: "2"         # pin to version 2; omit to follow default
tools:
  - type: web_search
    name: web
```

```bash
azd ai toolbox create agent-tools --from-file tools.yaml
```

Get the toolbox endpoint after creation:

```bash
ENDPOINT=$(azd ai toolbox show agent-tools -o json | jq -r .endpoint)
azd env set TOOLBOX_ENDPOINT "$ENDPOINT"
```

When `version` is omitted from a skill entry, the toolbox resolves the skill's `default_version` at read time. If the skill is updated (`azd ai skill update`), agents on the consumer endpoint pick up the new content without a toolbox republish.

## Recipe: remove skill from toolbox

```bash
# Remove one or more skills (one new version)
azd ai toolbox skill remove agent-tools my-skill --force

# Promote
azd ai toolbox publish agent-tools <new-version>
```

Removing the last skill is allowed (the toolbox can still have connections and tools).

## Versioning behavior

- Each `skill add` / `skill remove` creates a new immutable toolbox version (default unchanged until `publish`).
- Skill references without a pinned version follow the skill's `default_version` at read time.
- Skill references with a pinned version (`skill@2`) stay on that version regardless of skill updates.
- To rollback: `azd ai toolbox publish <toolbox> <previous-version>`.

## How skills appear at runtime (raw MCP protocol)

Skills are exposed through the MCP **resources** protocol (not `tools/list`):

- `resources/list` advertises each skill as a resource with a `skill://<name>/SKILL.md` URI (name + description).
- `resources/list` also exposes `skill://index.json` — a discovery index listing every skill on the toolbox version with URLs to read each skill's `SKILL.md` and (for multi-file skills) its ZIP archive.
- `resources/read` with a `skill://` URI retrieves the full `SKILL.md` body on demand.

### Raw MCP call examples

```bash
# Get a bearer token
TOK=$(az account get-access-token --resource "https://ai.azure.com" --query accessToken -o tsv)

# Foundry project endpoint (no trailing slash)
PE="<FOUNDRY_PROJECT_ENDPOINT>"
URL="$PE/toolboxes/<toolbox>/mcp?api-version=v1"
# List available skills (resources/list)
curl -s -X POST "$URL" \
  -H "Authorization: Bearer $TOK" \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","id":1,"method":"resources/list","params":{}}'

# Read skill index
curl -s -X POST "$URL" \
  -H "Authorization: Bearer $TOK" \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","id":2,"method":"resources/read","params":{"uri":"skill://index.json"}}'

# Read a specific skill's content
curl -s -X POST "$URL" \
  -H "Authorization: Bearer $TOK" \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","id":3,"method":"resources/read","params":{"uri":"skill://my-skill/SKILL.md"}}'
```

## Skill + Tool Search interaction

When `toolbox_search_preview` is enabled, regular tools are hidden from `tools/list` and discovered via `tool_search`. Skills remain in `resources/list` regardless of this setting — they are not affected by Tool Search.

## References

- [skill-manage.md](skill-manage.md) — create, version, and manage skills
- [skill-attach.md](skill-attach.md) — consume skills in agent code (direct download or toolbox MCP), choosing an approach

<#
.SYNOPSIS
    Canvas-First Entry check for the Foundry "create a new agent" flow.
.DESCRIPTION
    Reports -- using the same [OK]/[WARN]/[ACTION] convention as verify-environment --
    whether the canvas-first gate applies before scaffolding a new hosted agent in the
    GitHub Copilot app, from two deterministic facts:
      * copilot_app      - is AI_AGENT=github_copilot_app_agent?
      * canvas_installed - is the Foundry Agent Canvas (foundry-agent-canvas)
                           installed in the project, user, or session location?

    Output lines are prefixed with [OK], [WARN], or [ACTION].
    Exit code is 0 when the gate does not apply (not in the app, or the canvas is
    not installed), 1 when the canvas is installed and an [ACTION] is required.

    NOTE: whether the canvas is *open* is runtime UI state the agent reads from the
    message <canvas-context> (look for canvas="agent-builder"); a script cannot see it.
.EXAMPLE
    ./check-canvas-entry.ps1
#>

$ErrorActionPreference = "Stop"
$ext = "foundry-agent-canvas"

if ($env:AI_AGENT -ne "github_copilot_app_agent") {
    Write-Output "[OK] Not running in the GitHub Copilot app (AI_AGENT is not github_copilot_app_agent)."
    Write-Output "[OK] Canvas-first gate does not apply -- continue with the normal create workflow."
    exit 0
}

Write-Output "[OK] GitHub Copilot app detected (AI_AGENT=github_copilot_app_agent)."

# Candidate install locations: project (repo), user, session.
$dirs = @()
try { $root = (& git rev-parse --show-toplevel 2>$null); if ($root) { $dirs += (Join-Path $root ".github/extensions/$ext") } } catch {}
$dirs += (Join-Path (Get-Location) ".github/extensions/$ext")
$homeDir = if ($env:USERPROFILE) { $env:USERPROFILE } else { $HOME }
if ($homeDir) { $dirs += (Join-Path $homeDir ".copilot/extensions/$ext") }
if ($homeDir -and $env:COPILOT_AGENT_SESSION_ID) {
    $dirs += (Join-Path $homeDir ".copilot/session-state/$($env:COPILOT_AGENT_SESSION_ID)/extensions/$ext")
}

$installedAt = $null
foreach ($d in $dirs) {
    if (Test-Path (Join-Path $d "extension.mjs")) { $installedAt = $d; break }
}

if (-not $installedAt) {
    Write-Output "[WARN] Foundry Agent Canvas is not installed -- canvas-first gate does not apply; continue with the normal create workflow."
    exit 0
}

Write-Output "[OK] Foundry Agent Canvas is installed."
Write-Output '[ACTION] Canvas-first gate applies.'
exit 1

#!/usr/bin/env bash
# check-canvas-entry.sh
# Canvas-First Entry check for the Foundry "create a new agent" flow.
#
# Reports -- using the same [OK]/[WARN]/[ACTION] convention as verify-environment --
# whether the canvas-first gate applies before scaffolding a new hosted agent in the
# GitHub Copilot app, from two deterministic facts:
#   * copilot_app      - is AI_AGENT=github_copilot_app_agent?
#   * canvas_installed - is the Foundry Agent Canvas (foundry-agent-canvas)
#                        installed in the project, user, or session location?
#
# Output: summary lines, each prefixed with [OK], [WARN], or [ACTION].
# Exit code: 0 when the gate does not apply (not in the app, or the canvas is not
# installed), 1 when the canvas is installed and an [ACTION] is required.
#
# NOTE: whether the canvas is *open* is runtime UI state the agent reads from the
# message <canvas-context> (look for canvas="agent-builder"); a script cannot see it.

set -uo pipefail
ext="foundry-agent-canvas"

if [ "${AI_AGENT:-}" != "github_copilot_app_agent" ]; then
  echo "[OK] Not running in the GitHub Copilot app (AI_AGENT is not github_copilot_app_agent)."
  echo "[OK] Canvas-first gate does not apply -- continue with the normal create workflow."
  exit 0
fi

echo "[OK] GitHub Copilot app detected (AI_AGENT=github_copilot_app_agent)."

# Candidate install locations: project (repo), user, session.
dirs=()
root="$(git rev-parse --show-toplevel 2>/dev/null || true)"
[ -n "$root" ] && dirs+=("$root/.github/extensions/$ext")
dirs+=("$PWD/.github/extensions/$ext")
home_dir="${HOME:-${USERPROFILE:-}}"
[ -n "$home_dir" ] && dirs+=("$home_dir/.copilot/extensions/$ext")
[ -n "$home_dir" ] && [ -n "${COPILOT_AGENT_SESSION_ID:-}" ] && \
  dirs+=("$home_dir/.copilot/session-state/$COPILOT_AGENT_SESSION_ID/extensions/$ext")

installed_at=""
for d in "${dirs[@]}"; do
  if [ -f "$d/extension.mjs" ]; then installed_at="$d"; break; fi
done

if [ -z "$installed_at" ]; then
  echo "[WARN] Foundry Agent Canvas is not installed -- canvas-first gate does not apply; continue with the normal create workflow."
  exit 0
fi

echo "[OK] Foundry Agent Canvas is installed."
echo '[ACTION] Canvas-first gate applies.'
exit 1

<#
.SYNOPSIS
    Resolves a Foundry project ARM resource ID from a Foundry project endpoint.
.DESCRIPTION
    Uses the endpoint only to obtain lookup keys for Azure CLI queries. The
    resource ID printed by this script is always the `id` returned by Azure,
    never a locally constructed ARM resource ID.
.EXAMPLE
    ./resolve-project-id.ps1 -Endpoint "https://my-account.services.ai.azure.com/api/projects/my-project"
.EXAMPLE
    ./resolve-project-id.ps1 -Endpoint "https://my-account.services.ai.azure.com/api/projects/my-project" -Output json
#>

[CmdletBinding()]
param(
    [Parameter(Mandatory = $true)]
    [string]$Endpoint,

    [string]$Subscription,

    [string]$ResourceGroup,

    [string]$AccountName,

    [string]$ProjectName,

    [ValidateSet("id", "json")]
    [string]$Output = "id"
)

$ErrorActionPreference = "Stop"

function Stop-Fatal {
    param([string]$Message)
    [Console]::Error.WriteLine("[ERROR] $Message")
    exit 1
}

function Normalize-Endpoint {
    param([string]$Value)
    if (-not $Value) { return "" }
    return $Value.Trim().TrimEnd("/")
}

function Add-SubscriptionArg {
    param([string[]]$CommandArgs)
    if ($Subscription) {
        return $CommandArgs + @("--subscription", $Subscription)
    }
    return $CommandArgs
}

function Invoke-AzJson {
    param([string[]]$CommandArgs)
    $raw = & az @CommandArgs 2>&1
    if ($LASTEXITCODE -ne 0) {
        throw "az $($CommandArgs -join ' ') failed: $($raw -join "`n")"
    }
    if (-not $raw) { return $null }
    return (($raw -join "`n") | ConvertFrom-Json -ErrorAction Stop)
}

function Get-ProjectEndpoints {
    param($Project)
    $values = @()
    if ($Project -and $Project.properties -and $Project.properties.endpoints) {
        foreach ($property in $Project.properties.endpoints.PSObject.Properties) {
            if ($property.Value -is [string] -and $property.Value) {
                $values += (Normalize-Endpoint $property.Value)
            }
        }
    }
    return $values
}

function Endpoint-MatchesProject {
    param($Project, [string]$ExpectedEndpoint)
    foreach ($candidate in (Get-ProjectEndpoints $Project)) {
        if ($candidate -eq $ExpectedEndpoint) {
            return $true
        }
    }
    return $false
}

if (-not (Get-Command az -ErrorAction SilentlyContinue)) {
    Stop-Fatal "Azure CLI 'az' was not found on PATH."
}

$normalizedEndpoint = Normalize-Endpoint $Endpoint
try {
    $endpointUri = [System.Uri]$normalizedEndpoint
} catch {
    Stop-Fatal "Endpoint is not a valid URI: $Endpoint"
}

if (-not $endpointUri.Scheme.StartsWith("http")) {
    Stop-Fatal "Endpoint must be an http or https URI."
}

if (-not $ProjectName) {
    $segments = @($endpointUri.AbsolutePath.Trim("/").Split("/", [System.StringSplitOptions]::RemoveEmptyEntries))
    for ($i = 0; $i -lt $segments.Count; $i++) {
        if ($segments[$i] -ieq "projects" -and ($i + 1) -lt $segments.Count) {
            $ProjectName = [System.Uri]::UnescapeDataString($segments[$i + 1])
            break
        }
    }
}

if (-not $ProjectName) {
    Stop-Fatal "Could not read the project name from the endpoint path. Re-run with -ProjectName."
}

if (-not $AccountName) {
    $hostParts = @($endpointUri.Host.Split("."))
    if ($hostParts.Count -gt 0 -and $endpointUri.Host.EndsWith(".services.ai.azure.com", [System.StringComparison]::OrdinalIgnoreCase)) {
        $AccountName = $hostParts[0]
    }
}

if (-not $AccountName) {
    Stop-Fatal "Could not read the account name from the endpoint host. Re-run with -AccountName."
}

if (-not $ResourceGroup) {
    try {
        $accounts = Invoke-AzJson (Add-SubscriptionArg @("cognitiveservices", "account", "list", "-o", "json"))
    } catch {
        Stop-Fatal $_.Exception.Message
    }

    $matches = @($accounts | Where-Object {
        ($_.name -ieq $AccountName) -or
        ($_.properties.customSubDomainName -ieq $AccountName)
    })

    if ($matches.Count -eq 0) {
        Stop-Fatal "Could not find a Cognitive Services account matching '$AccountName'. Re-run with -ResourceGroup and -AccountName if the endpoint uses a custom host."
    }

    if ($matches.Count -gt 1) {
        $choices = ($matches | ForEach-Object { "$($_.resourceGroup)/$($_.name)" }) -join ", "
        Stop-Fatal "Multiple accounts matched '$AccountName': $choices. Re-run with -ResourceGroup."
    }

    $ResourceGroup = $matches[0].resourceGroup
    $AccountName = $matches[0].name
}

$project = $null
try {
    $project = Invoke-AzJson (Add-SubscriptionArg @(
        "cognitiveservices", "account", "project", "show",
        "-g", $ResourceGroup,
        "-n", $AccountName,
        "--project-name", $ProjectName,
        "-o", "json"
    ))
} catch {
    try {
        $projects = Invoke-AzJson (Add-SubscriptionArg @(
            "cognitiveservices", "account", "project", "list",
            "-g", $ResourceGroup,
            "-n", $AccountName,
            "-o", "json"
        ))
        $project = @($projects | Where-Object { Endpoint-MatchesProject $_ $normalizedEndpoint }) | Select-Object -First 1
    } catch {
        Stop-Fatal $_.Exception.Message
    }
}

if (-not $project) {
    Stop-Fatal "Could not resolve a Foundry project for endpoint '$normalizedEndpoint'."
}

$projectEndpoints = @(Get-ProjectEndpoints $project)
if ($projectEndpoints.Count -gt 0 -and -not (Endpoint-MatchesProject $project $normalizedEndpoint)) {
    Stop-Fatal "Resolved project endpoint metadata did not match '$normalizedEndpoint'."
}

if (-not $project.id) {
    Stop-Fatal "Azure returned a project object without an id."
}

if ($Output -eq "json") {
    [ordered]@{
        id = $project.id
        endpoint = if ($projectEndpoints.Count -gt 0) { $projectEndpoints[0] } else { $normalizedEndpoint }
        resourceGroup = $ResourceGroup
        accountName = $AccountName
        projectName = $ProjectName
    } | ConvertTo-Json -Depth 5
} else {
    Write-Output $project.id
}

#!/usr/bin/env bash
# resolve-project-id.sh
# Resolves a Foundry project ARM resource ID from a Foundry project endpoint.
# The endpoint is used only for Azure lookup keys; the printed ID is the `id`
# returned by Azure CLI, not a locally constructed ARM resource ID.
#
# Usage:
#   ./resolve-project-id.sh --endpoint "https://my-account.services.ai.azure.com/api/projects/my-project"
#   ./resolve-project-id.sh --endpoint "https://my-account.services.ai.azure.com/api/projects/my-project" --output json

set -uo pipefail

ENDPOINT=""
SUBSCRIPTION=""
RESOURCE_GROUP=""
ACCOUNT_NAME=""
PROJECT_NAME=""
OUTPUT="id"
TEMP_FILES=()

cleanup() {
  if [ "${#TEMP_FILES[@]}" -gt 0 ]; then
    rm -f "${TEMP_FILES[@]}"
  fi
}
trap cleanup EXIT

usage() {
  cat <<'EOF'
Usage: resolve-project-id.sh --endpoint <foundry-project-endpoint> [options]

Options:
  -e, --endpoint <url>          Foundry project endpoint. Required.
      --subscription <id>       Azure subscription ID or name.
  -g, --resource-group <name>   Resource group for the Foundry account.
  -n, --account-name <name>     Foundry account name.
      --project-name <name>     Foundry project name.
  -o, --output <id|json>        Output format. Default: id.
  -h, --help                    Show this help.
EOF
}

fatal() {
  echo "[ERROR] $1" >&2
  exit 1
}

while [ "$#" -gt 0 ]; do
  case "$1" in
    -e|--endpoint)
      [ "$#" -ge 2 ] || fatal "$1 requires a value."
      ENDPOINT="$2"
      shift 2
      ;;
    --subscription)
      [ "$#" -ge 2 ] || fatal "$1 requires a value."
      SUBSCRIPTION="$2"
      shift 2
      ;;
    -g|--resource-group)
      [ "$#" -ge 2 ] || fatal "$1 requires a value."
      RESOURCE_GROUP="$2"
      shift 2
      ;;
    -n|--account-name)
      [ "$#" -ge 2 ] || fatal "$1 requires a value."
      ACCOUNT_NAME="$2"
      shift 2
      ;;
    --project-name)
      [ "$#" -ge 2 ] || fatal "$1 requires a value."
      PROJECT_NAME="$2"
      shift 2
      ;;
    -o|--output)
      [ "$#" -ge 2 ] || fatal "$1 requires a value."
      OUTPUT="$2"
      shift 2
      ;;
    -h|--help)
      usage
      exit 0
      ;;
    *)
      fatal "Unknown argument: $1"
      ;;
  esac
done

[ -n "$ENDPOINT" ] || fatal "--endpoint is required."
[ "$OUTPUT" = "id" ] || [ "$OUTPUT" = "json" ] || fatal "--output must be 'id' or 'json'."

command -v az >/dev/null 2>&1 || fatal "Azure CLI 'az' was not found on PATH."
command -v python3 >/dev/null 2>&1 || fatal "python3 was not found on PATH."

PARSED_ENDPOINT="$(
  python3 - "$ENDPOINT" "$ACCOUNT_NAME" "$PROJECT_NAME" <<'PY'
import json
import sys
from urllib.parse import unquote, urlparse

endpoint = (sys.argv[1] or "").strip().rstrip("/")
account_name = sys.argv[2] or ""
project_name = sys.argv[3] or ""

parsed = urlparse(endpoint)
if parsed.scheme not in ("http", "https") or not parsed.netloc:
    print("Endpoint must be an http or https URI.", file=sys.stderr)
    raise SystemExit(1)

if not project_name:
    parts = [unquote(p) for p in parsed.path.strip("/").split("/") if p]
    for index, part in enumerate(parts):
        if part.lower() == "projects" and index + 1 < len(parts):
            project_name = parts[index + 1]
            break

if not account_name:
    host = parsed.hostname or ""
    suffix = ".services.ai.azure.com"
    if host.lower().endswith(suffix):
        account_name = host[:-len(suffix)]

print(json.dumps({
    "endpoint": endpoint,
    "accountName": account_name,
    "projectName": project_name,
}))
PY
)" || fatal "Could not parse Foundry project endpoint."

NORMALIZED_ENDPOINT="$(python3 -c 'import json,sys; print(json.loads(sys.stdin.read())["endpoint"])' <<<"$PARSED_ENDPOINT")"
if [ -z "$ACCOUNT_NAME" ]; then
  ACCOUNT_NAME="$(python3 -c 'import json,sys; print(json.loads(sys.stdin.read())["accountName"])' <<<"$PARSED_ENDPOINT")"
fi
if [ -z "$PROJECT_NAME" ]; then
  PROJECT_NAME="$(python3 -c 'import json,sys; print(json.loads(sys.stdin.read())["projectName"])' <<<"$PARSED_ENDPOINT")"
fi

[ -n "$ACCOUNT_NAME" ] || fatal "Could not read the account name from the endpoint host. Re-run with --account-name."
[ -n "$PROJECT_NAME" ] || fatal "Could not read the project name from the endpoint path. Re-run with --project-name."

add_subscription_arg() {
  if [ -n "$SUBSCRIPTION" ]; then
    printf '%s\n' "--subscription" "$SUBSCRIPTION"
  fi
}

run_az_json() {
  local stderr_file
  stderr_file="$(mktemp)"
  local output
  if output="$(az "$@" 2>"$stderr_file")"; then
    rm -f "$stderr_file"
    printf '%s' "$output"
    return 0
  fi
  local error_text
  error_text="$(cat "$stderr_file")"
  rm -f "$stderr_file"
  echo "$error_text" >&2
  return 1
}

if [ -z "$RESOURCE_GROUP" ]; then
  AZ_ARGS=(cognitiveservices account list -o json)
  while IFS= read -r arg; do
    [ -n "$arg" ] && AZ_ARGS+=("$arg")
  done < <(add_subscription_arg)

  ACCOUNTS_JSON="$(run_az_json "${AZ_ARGS[@]}")" || fatal "Failed to list Cognitive Services accounts."
  ACCOUNTS_FILE="$(mktemp)"
  TEMP_FILES+=("$ACCOUNTS_FILE")
  printf '%s' "$ACCOUNTS_JSON" >"$ACCOUNTS_FILE"
  MATCHED_ACCOUNT="$(
    ACCOUNT_NAME="$ACCOUNT_NAME" python3 - "$ACCOUNTS_FILE" <<'PY'
import json
import os
import sys

target = os.environ["ACCOUNT_NAME"].lower()
with open(sys.argv[1], encoding="utf-8") as handle:
    accounts = json.load(handle)
matches = []
for account in accounts:
    name = (account.get("name") or "")
    custom = ((account.get("properties") or {}).get("customSubDomainName") or "")
    if name.lower() == target or custom.lower() == target:
        matches.append(account)

if not matches:
    print(f"Could not find a Cognitive Services account matching '{os.environ['ACCOUNT_NAME']}'.", file=sys.stderr)
    raise SystemExit(1)
if len(matches) > 1:
    choices = ", ".join(f"{m.get('resourceGroup')}/{m.get('name')}" for m in matches)
    print(f"Multiple accounts matched '{os.environ['ACCOUNT_NAME']}': {choices}. Re-run with --resource-group.", file=sys.stderr)
    raise SystemExit(1)

print(json.dumps({
    "resourceGroup": matches[0].get("resourceGroup") or "",
    "accountName": matches[0].get("name") or "",
}))
PY
  )" || fatal "Failed to resolve the Foundry account resource group."

  RESOURCE_GROUP="$(python3 -c 'import json,sys; print(json.loads(sys.stdin.read())["resourceGroup"])' <<<"$MATCHED_ACCOUNT")"
  ACCOUNT_NAME="$(python3 -c 'import json,sys; print(json.loads(sys.stdin.read())["accountName"])' <<<"$MATCHED_ACCOUNT")"
fi

PROJECT_JSON=""
AZ_SHOW_ARGS=(
  cognitiveservices account project show
  -g "$RESOURCE_GROUP"
  -n "$ACCOUNT_NAME"
  --project-name "$PROJECT_NAME"
  -o json
)
while IFS= read -r arg; do
  [ -n "$arg" ] && AZ_SHOW_ARGS+=("$arg")
done < <(add_subscription_arg)

if ! PROJECT_JSON="$(run_az_json "${AZ_SHOW_ARGS[@]}")"; then
  AZ_LIST_ARGS=(
    cognitiveservices account project list
    -g "$RESOURCE_GROUP"
    -n "$ACCOUNT_NAME"
    -o json
  )
  while IFS= read -r arg; do
    [ -n "$arg" ] && AZ_LIST_ARGS+=("$arg")
  done < <(add_subscription_arg)

  PROJECTS_JSON="$(run_az_json "${AZ_LIST_ARGS[@]}")" || fatal "Failed to list Foundry projects."
  PROJECTS_FILE="$(mktemp)"
  TEMP_FILES+=("$PROJECTS_FILE")
  printf '%s' "$PROJECTS_JSON" >"$PROJECTS_FILE"
  PROJECT_JSON="$(
    NORMALIZED_ENDPOINT="$NORMALIZED_ENDPOINT" python3 - "$PROJECTS_FILE" <<'PY'
import json
import os
import sys

expected = os.environ["NORMALIZED_ENDPOINT"].rstrip("/")
with open(sys.argv[1], encoding="utf-8") as handle:
    projects = json.load(handle)

def endpoints(project):
    values = ((project.get("properties") or {}).get("endpoints") or {}).values()
    return [value.rstrip("/") for value in values if isinstance(value, str) and value]

for project in projects:
    if expected in endpoints(project):
        print(json.dumps(project))
        break
else:
    print(f"Could not find a Foundry project matching endpoint '{expected}'.", file=sys.stderr)
    raise SystemExit(1)
PY
  )" || fatal "Failed to resolve the Foundry project from endpoint metadata."
fi

PROJECT_JSON="$PROJECT_JSON" \
NORMALIZED_ENDPOINT="$NORMALIZED_ENDPOINT" \
RESOURCE_GROUP="$RESOURCE_GROUP" \
ACCOUNT_NAME="$ACCOUNT_NAME" \
PROJECT_NAME="$PROJECT_NAME" \
OUTPUT="$OUTPUT" \
python3 - <<'PY'
import json
import os

project = json.loads(os.environ["PROJECT_JSON"])
expected = os.environ["NORMALIZED_ENDPOINT"].rstrip("/")
endpoint_values = ((project.get("properties") or {}).get("endpoints") or {}).values()
endpoints = [value.rstrip("/") for value in endpoint_values if isinstance(value, str) and value]

if endpoints and expected not in endpoints:
    print(f"[ERROR] Resolved project endpoint metadata did not match '{expected}'.", file=__import__("sys").stderr)
    raise SystemExit(1)

resource_id = project.get("id")
if not resource_id:
    print("[ERROR] Azure returned a project object without an id.", file=__import__("sys").stderr)
    raise SystemExit(1)

if os.environ["OUTPUT"] == "json":
    print(json.dumps({
        "id": resource_id,
        "endpoint": endpoints[0] if endpoints else expected,
        "resourceGroup": os.environ["RESOURCE_GROUP"],
        "accountName": os.environ["ACCOUNT_NAME"],
        "projectName": os.environ["PROJECT_NAME"],
    }, indent=2))
else:
    print(resource_id)
PY

<#
.SYNOPSIS
    Verifies the local environment for creating a hosted Foundry agent with `azd ai`.
.DESCRIPTION
    Runs all the read-only checks in one pass and prints a single concise summary,
    so the agent does not have to run (and reason over) each azd command separately.

    Output lines are prefixed with [OK], [WARN], or [ACTION].
    Exit code is 0 when no blocking actions remain, 1 when at least one [ACTION] is required.
.EXAMPLE
    ./verify-environment.ps1
#>

$ErrorActionPreference = "Stop"
$actionRequired = $false

function Note-Ok     { param([string]$m) Write-Output "[OK] $m" }
function Note-Warn   { param([string]$m) Write-Output "[WARN] $m" }
function Note-Action { param([string]$m) Write-Output "[ACTION] $m"; $script:actionRequired = $true }

function Get-AzdJson {
    param([string[]]$AzdArgs)
    try {
        $raw = & azd @AzdArgs 2>$null
        if (-not $raw) { return $null }
        return ($raw | ConvertFrom-Json -ErrorAction Stop)
    } catch {
        return $null
    }
}

# Refresh PATH to pick up recently-installed tools (e.g. azd installed in same session)
$env:Path = [System.Environment]::GetEnvironmentVariable("Path", "Machine") + ";" + [System.Environment]::GetEnvironmentVariable("Path", "User")

function Add-CommandFallbackPath {
    param(
        [string] $CommandName,
        [string[]] $Directories
    )

    if (Get-Command $CommandName -ErrorAction SilentlyContinue) {
        return [pscustomobject]@{ Found = $true; AddedPath = $null }
    }

    foreach ($dir in $Directories) {
        if (-not $dir) { continue }
        foreach ($ext in @(".exe", ".cmd", ".bat")) {
            $candidate = Join-Path $dir "$CommandName$ext"
            if (Test-Path $candidate) {
                $env:Path = "$dir;$env:Path"
                return [pscustomobject]@{ Found = $true; AddedPath = $dir }
            }
        }
    }

    return [pscustomobject]@{ Found = [bool](Get-Command $CommandName -ErrorAction SilentlyContinue); AddedPath = $null }
}

function Test-AzdAuthLoggedIn {
    $raw = ""
    try {
        $raw = (& azd auth login --check-status 2>&1) -join "`n"
    } catch {
        $raw = $_ | Out-String
    }
    $authExit = $LASTEXITCODE

    if ($raw -match "(?i)(not\s+logged\s+in|not\s+authenticated|no\s+account|login\s+required|please\s+run.*azd\s+auth\s+login|run.*azd\s+auth\s+login|expired)") {
        return $false
    }

    if ($raw -match "(?i)(logged\s+in|authenticated|already\s+logged\s+in)") {
        return $true
    }

    # Unrecognized output -- fall back to exit code
    return ($authExit -eq 0)
}

# 1. Required CLIs
# Check PATH first, then probe common install locations (winget, MSI, chocolatey)
$azdCommand = Add-CommandFallbackPath "azd" @(
    "$env:LOCALAPPDATA\Programs\Azure Dev CLI",
    "$env:ProgramFiles\Azure Dev CLI",
    "${env:ProgramFiles(x86)}\Azure Dev CLI",
    "$env:USERPROFILE\.azd\bin"
)
$azdInstalled = $azdCommand.Found
if ($azdCommand.AddedPath) {
    Note-Warn "azd found at '$($azdCommand.AddedPath)' but was not on PATH. Added automatically for this session."
}
if (-not $azdInstalled) {
    Note-Action "Azure Developer CLI (azd) is not installed. Install it from https://aka.ms/azd-install, then re-run."
}

$azCommand = Add-CommandFallbackPath "az" @(
    "$env:ProgramFiles\Microsoft SDKs\Azure\CLI2\wbin",
    "${env:ProgramFiles(x86)}\Microsoft SDKs\Azure\CLI2\wbin"
)
$azInstalled = $azCommand.Found
if ($azCommand.AddedPath) {
    Note-Warn "az found at '$($azCommand.AddedPath)' but was not on PATH. Added automatically for this session."
}
if (-not $azInstalled) {
    Note-Action "Azure CLI (az) is not installed. Install it from https://aka.ms/installazurecli, then re-run."
}

if (-not $azdInstalled -or -not $azInstalled) {
    Write-Output ""
    Write-Output "Summary: CLI missing -- cannot continue."
    exit 1
}

$verJson = Get-AzdJson @("version", "--output", "json")
$azdVersion = if ($verJson -and $verJson.azd -and $verJson.azd.version) { $verJson.azd.version } else { "unknown" }
Note-Ok "azd installed (version $azdVersion)."

try {
    $azVersionRaw = (& az version --query '"azure-cli"' -o tsv 2>$null) -join "`n"
} catch {
    $azVersionRaw = ""
}
$azVersion = if ($azVersionRaw) { $azVersionRaw.Trim() } else { "unknown" }
Note-Ok "Azure CLI installed (version $azVersion)."

# 2. Required azd extensions
try {
    $extRaw = (& azd extension list --installed --output json 2>$null) -join "`n"
} catch {
    $extRaw = ""
}
foreach ($ext in @("azure.ai.agents", "azure.ai.projects", "microsoft.foundry")) {
    if ($extRaw -match [regex]::Escape($ext)) {
        Note-Ok "Extension '$ext' is installed."
    } else {
        Note-Action "Extension '$ext' is missing. Run: azd extension install $ext"
    }
}

# 3. Auth status
if (Test-AzdAuthLoggedIn) {
    Note-Ok "Logged in to azd."
} else {
    Note-Action "Not logged in to azd. Ask the user to run 'azd auth login' (it opens a browser; never run it for them)."
}

try {
    $azAccountRaw = (& az account show --output json 2>$null) -join "`n"
} catch {
    $azAccountRaw = ""
}
if (-not $azAccountRaw) {
    Note-Action "Not logged in to Azure CLI. Ask the user to run 'az login' (it opens a browser; never run it for them)."
} else {
    try {
        $azAccount = $azAccountRaw | ConvertFrom-Json -ErrorAction Stop
        $state = if ($azAccount.PSObject.Properties.Name -contains "state") { $azAccount.state } else { "" }
        if ($state -and $state -ne "Enabled") {
            Note-Action "Azure CLI active subscription state is '$state'. Ask the user to select an enabled subscription with 'az account set --subscription <id>'."
        } else {
            $subName = if ($azAccount.PSObject.Properties.Name -contains "name" -and $azAccount.name) { $azAccount.name } else { "unknown" }
            Note-Ok "Azure CLI logged in (subscription: $subName)."
        }
    } catch {
        Note-Action "Unable to verify Azure CLI login status. Ask the user to run 'az login' and re-run this script."
    }
}

if ($actionRequired) {
    Write-Output ""
    Write-Output "Summary: action required -- resolve the [ACTION] items above before continuing."
    exit 1
}

# 4. Foundry project endpoint (optional at this stage)
# Short-circuit when there's no azd project in cwd: `azd ai project show` / `agent show`
# would just return nothing after a ~3s subprocess each.
if (-not (Test-Path "azure.yaml")) {
    Note-Warn "No Foundry project endpoint set yet. A new project will be created at provision/deploy time, or supply an existing project resource ID."
    Note-Ok "No agent deployed yet. Proceed with create."
} else {
    $projectJson = Get-AzdJson @("ai", "project", "show", "--output", "json")
    $endpoint = $null
    if ($projectJson) {
        foreach ($k in @("endpoint", "projectEndpoint", "aiProjectEndpoint")) {
            if ($projectJson.PSObject.Properties.Name -contains $k -and $projectJson.$k) {
                $endpoint = $projectJson.$k
                break
            }
        }
    }
    if ($endpoint) {
        Note-Ok "Foundry project endpoint configured: $endpoint"
    } else {
        Note-Warn "No Foundry project endpoint set yet. A new project will be created at provision/deploy time, or supply an existing project resource ID."
    }

    # 5. Agent deployment status
    $agentJson = Get-AzdJson @("ai", "agent", "show", "--output", "json")
    if ($agentJson) {
        $status = if ($agentJson.PSObject.Properties.Name -contains "status" -and $agentJson.status) { $agentJson.status } else { "unknown" }
        switch ($status) {
            { $_ -in @("active", "deployed") } { Note-Ok "An agent is already deployed (status: $status). Skip to deploy.md to redeploy, or tools to add a tool." }
            "not_deployed"                     { Note-Ok "No agent deployed yet (status: not_deployed). Proceed with create." }
            default                            { Note-Warn "Agent status: $status." }
        }
    } else {
        Note-Ok "No agent deployed yet. Proceed with create."
    }
}

Write-Output ""
if ($actionRequired) {
    Write-Output "Summary: action required -- resolve the [ACTION] items above before continuing."
    exit 1
} else {
    Write-Output "Summary: environment ready for 'azd ai' hosted-agent creation."
    exit 0
}

#!/usr/bin/env bash
# verify-environment.sh
# Verifies the local environment for creating a hosted Foundry agent with `azd ai`.
# Runs all the read-only checks in one pass and prints a single concise summary,
# so the agent does not have to run (and reason over) each azd command separately.
#
# Usage:
#   ./verify-environment.sh
#
# Output: human-readable summary lines, each prefixed with [OK], [WARN], or [ACTION].
# Exit code: 0 if no blocking actions, 1 if at least one [ACTION] is required.

set -uo pipefail

ACTION_REQUIRED=0

note_ok()     { echo "[OK] $1"; }
note_warn()   { echo "[WARN] $1"; }
note_action() { echo "[ACTION] $1"; ACTION_REQUIRED=1; }

# Refresh PATH to pick up recently-installed tools (e.g. azd installed in same session)
if [ -f /etc/environment ]; then
  # shellcheck disable=SC1091
  . /etc/environment 2>/dev/null || true
fi
hash -r 2>/dev/null || true

# 1. Required CLIs
AZD_AVAILABLE=1
AZ_AVAILABLE=1

if ! command -v azd >/dev/null 2>&1; then
  note_action "Azure Developer CLI (azd) is not installed. Install it from https://aka.ms/azd-install, then re-run."
  AZD_AVAILABLE=0
fi

if ! command -v az >/dev/null 2>&1; then
  note_action "Azure CLI (az) is not installed. Install it from https://aka.ms/installazurecli, then re-run."
  AZ_AVAILABLE=0
fi

if [ "$AZD_AVAILABLE" -eq 0 ] || [ "$AZ_AVAILABLE" -eq 0 ]; then
  echo ""
  echo "Summary: CLI missing -- cannot continue."
  exit 1
fi

AZD_VERSION="$(azd version --output json 2>/dev/null | python3 -c 'import json,sys; print(json.load(sys.stdin).get("azd",{}).get("version","unknown"))' 2>/dev/null || echo unknown)"
note_ok "azd installed (version ${AZD_VERSION})."

AZ_VERSION="$(az version --query '"azure-cli"' -o tsv 2>/dev/null || echo unknown)"
note_ok "Azure CLI installed (version ${AZ_VERSION})."

# 2. Required azd extensions
EXT_JSON="$(azd extension list --installed --output json 2>/dev/null || echo '[]')"
for ext in azure.ai.agents azure.ai.projects microsoft.foundry; do
  if printf '%s' "$EXT_JSON" | grep -q "$ext"; then
    note_ok "Extension '$ext' is installed."
  else
    note_action "Extension '$ext' is missing. Run: azd extension install $ext"
  fi
done

# 3. Auth status
AZD_AUTH_OUTPUT="$(azd auth login --check-status 2>&1)"; AZD_AUTH_EXIT=$?
if printf '%s' "$AZD_AUTH_OUTPUT" | grep -Eiq '(not[[:space:]]+logged[[:space:]]+in|not[[:space:]]+authenticated|no[[:space:]]+account|login[[:space:]]+required|please[[:space:]]+run.*azd[[:space:]]+auth[[:space:]]+login|run.*azd[[:space:]]+auth[[:space:]]+login|expired)'; then
  note_action "Not logged in to azd. Ask the user to run 'azd auth login' (it opens a browser; never run it for them)."
elif printf '%s' "$AZD_AUTH_OUTPUT" | grep -Eiq '(logged[[:space:]]+in|authenticated|already[[:space:]]+logged[[:space:]]+in)'; then
  note_ok "Logged in to azd."
elif [ "$AZD_AUTH_EXIT" -eq 0 ]; then
  # Unrecognized output -- fall back to exit code
  note_ok "Logged in to azd."
else
  note_action "Unable to verify azd auth status. Ask the user to run 'azd auth login' and re-run this script."
fi

AZ_ACCOUNT_JSON="$(az account show --output json 2>/dev/null || true)"
if [ -z "$AZ_ACCOUNT_JSON" ]; then
  note_action "Not logged in to Azure CLI. Ask the user to run 'az login' (it opens a browser; never run it for them)."
else
  AZ_ACCOUNT_PARSED="$(printf '%s' "$AZ_ACCOUNT_JSON" | python3 -c 'import json,sys
try:
    d=json.load(sys.stdin)
except Exception:
    raise SystemExit(1)
if not isinstance(d, dict):
    raise SystemExit(1)
print((d.get("name") or "unknown").replace("\t", " "), d.get("state") or "", sep="\t")
' 2>/dev/null || true)"
  if [ -z "$AZ_ACCOUNT_PARSED" ]; then
    note_action "Unable to verify Azure CLI login status. Ask the user to run 'az login' and re-run this script."
  else
    IFS=$'\t' read -r AZ_SUB_NAME AZ_SUB_STATE <<< "$AZ_ACCOUNT_PARSED"
    AZ_SUB_STATE="${AZ_SUB_STATE//$'\r'/}"
    if [ -n "$AZ_SUB_STATE" ] && [ "$AZ_SUB_STATE" != "Enabled" ]; then
      note_action "Azure CLI active subscription state is '${AZ_SUB_STATE}'. Ask the user to select an enabled subscription with 'az account set --subscription <id>'."
    else
      note_ok "Azure CLI logged in (subscription: ${AZ_SUB_NAME:-unknown})."
    fi
  fi
fi

if [ "$ACTION_REQUIRED" -eq 1 ]; then
  echo ""
  echo "Summary: action required -- resolve the [ACTION] items above before continuing."
  exit 1
fi

# 4. Foundry project endpoint (optional at this stage)
# Short-circuit when there's no azd project in cwd: `azd ai project show` / `agent show`
# would just return nothing after a ~3s subprocess each.
if [ ! -f "azure.yaml" ]; then
  note_warn "No Foundry project endpoint set yet. A new project will be created at provision/deploy time, or supply an existing project resource ID."
  note_ok "No agent deployed yet. Proceed with create."
else
  PROJECT_JSON="$(azd ai project show --output json 2>/dev/null || echo '')"
  ENDPOINT=""
  if [ -n "$PROJECT_JSON" ]; then
    ENDPOINT="$(printf '%s' "$PROJECT_JSON" | python3 -c 'import json,sys
try:
    d=json.load(sys.stdin)
except Exception:
    print(""); raise SystemExit
if isinstance(d,dict):
    for k in ("endpoint","projectEndpoint","aiProjectEndpoint"):
        if d.get(k):
            print(d[k]); break
' 2>/dev/null)"
  fi
  if [ -n "$ENDPOINT" ]; then
    note_ok "Foundry project endpoint configured: ${ENDPOINT}"
  else
    note_warn "No Foundry project endpoint set yet. A new project will be created at provision/deploy time, or supply an existing project resource ID."
  fi

  # 5. Agent deployment status
  AGENT_JSON="$(azd ai agent show --output json 2>/dev/null || echo '')"
  if [ -n "$AGENT_JSON" ]; then
    STATUS="$(printf '%s' "$AGENT_JSON" | python3 -c 'import json,sys
try:
    d=json.load(sys.stdin)
except Exception:
    print("unknown"); raise SystemExit
print(d.get("status","unknown") if isinstance(d,dict) else "unknown")' 2>/dev/null)"
    case "$STATUS" in
      active|deployed) note_ok "An agent is already deployed (status: ${STATUS}). Skip to deploy.md to redeploy, or tools to add a tool." ;;
      not_deployed)    note_ok "No agent deployed yet (status: not_deployed). Proceed with create." ;;
      *)               note_warn "Agent status: ${STATUS}." ;;
    esac
  else
    note_ok "No agent deployed yet. Proceed with create."
  fi
fi

echo ""
if [ "$ACTION_REQUIRED" -eq 1 ]; then
  echo "Summary: action required -- resolve the [ACTION] items above before continuing."
  exit 1
else
  echo "Summary: environment ready for 'azd ai' hosted-agent creation."
  exit 0
fi

# Deploy a Foundry Agent

Provision Azure resources when needed, deploy the agent, and smoke-test it.

For **hosted agents** (custom container or code), use `azd deploy`. Prefer **direct code deployment through azd** (no Docker/ACR required): the agent's `azure.yaml` service block must contain `codeConfiguration:`, so `azd deploy` will use direct code deployment and zip the source and let Foundry build it. Use container/ACR deployment only when the agent truly needs a Dockerfile, custom system packages, or a pre-built image.

For **prompt agents** (LLM + instructions, no custom code), use the Foundry MCP `agent_update` tool.

## Quick Reference

| Property | Value |
|----------|-------|
| Hosted (recommended) | `azd provision` when needed, direct code deployment via `azd deploy` (`codeConfiguration` present), `azd ai agent invoke` |
| Hosted (container) | `azd provision` when needed, container/ACR deployment via `azd deploy` (requires Docker/Podman + ACR, no `codeConfiguration:` in the `azure.yaml` service block) |
| Prompt MCP | `agent_definition_schema_get`, `agent_update`, `agent_get`, `agent_delete` |
| Versioning | Each successful `azd deploy` creates an immutable agent version |
| Endpoint-only patch | `azd ai agent endpoint update` (no new version) |
| Local dev | [create-hosted](../create/create-hosted.md), [local-run](../create/references/local-run.md) |

## Hosted vs Prompt

- Shipping Python / .NET code -> **Hosted** (azd workflow below).
- Updating only model / instructions / tools -> **Prompt** (MCP workflow below).

## Deployment Method Selection -- Hosted agents

Before running `azd deploy`, inspect the agent's service block in `azure.yaml`.

| Service block state | Deployment path |
|------------------|-----------------|
| `codeConfiguration:` present | **Direct code deploy** through `azd deploy`; no Docker/ACR build. |
| No `codeConfiguration:` | **Container/ACR deploy** through `azd deploy`; builds/pushes an image or uses a pre-built `image:`. |

`codeConfiguration:` example in the `azure.yaml` service block:

```yaml
services:
  <agent-name>:
    host: azure.ai.agent
    codeConfiguration:
      runtime: python_3_13
      entryPoint: main.py
      dependencyResolution: remote_build
```

Default to direct code for standard hosted-agent code. If `azd deploy` prints `Packaging container` for an agent that does not need container-specific behavior, add or fix `codeConfiguration` and retry. Use the container path when the agent depends on Dockerfile behavior, system packages, or a pre-built image.

## Workflow -- Hosted agent (azd)

> Prerequisite: project scaffolded with `azd ai agent init`. If not, start at [create-hosted](../create/create-hosted.md).

### Step 1 -- Resolve azd environment

If the user provided an existing project endpoint, project ARM ID, or model deployment, set those values before deploy. Then verify the azd environment with `azd env get-values`.

```bash
azd env set AZURE_AI_PROJECT_ENDPOINT "<project-endpoint>"
azd env set AZURE_AI_PROJECT_ID "<project-arm-id>"
azd env set AZURE_AI_MODEL_DEPLOYMENT_NAME "<model-deployment-name>"
azd env get-values
```

Run:

```bash
azd ai project show --output json
azd ai agent show --output json
```

Branch on output: `not_deployed` -> Step 2. `active` / `deployed` -> redeploy (skip Step 2, go to Step 3). If `azd ai project show` fails with `missing_project_endpoint`, do Step 2 first -- `azd provision` will create the project.

> **Important:** Before deploy, also make sure the agent's `azure.yaml` service block and the azd environment are aligned with the user's provided configuration values.

### Step 2 -- Provision Azure resources (one-time per env)

> 🚦 **Project-selection gate.** If no foundry project endpoint is configured (not in the message, `azd env`, or `.env`) and the user hasn't asked to create one, stop and ask them to pick an existing foundry project or confirm creating a new one — don't silently select.

Skip `azd provision` when the user gave you an existing `AZURE_AI_PROJECT_ENDPOINT` or `FOUNDRY_PROJECT_ENDPOINT` and the workflow only needs to deploy the agent into that project.

Run provision only for new projects or real infrastructure changes:

```bash
azd provision --no-prompt
```

> Optional: run `azd provision --preview --no-prompt` first to preview the resource changes (a what-if) before applying them.
>
> Optional: add `--no-state` on a fresh azd environment to skip the existing-deployment check and provision faster; omit it when re-provisioning an existing one.

What this does:

- Creates the Foundry project (if not present) and supporting resources under `infra/`.
- Creates any connections/toolboxes declared as top-level `azure.ai.connection` / `azure.ai.toolbox` services (linked from the agent via `uses:`). Most agents instead reference an existing toolbox through a `TOOLBOX_<NAME>_MCP_ENDPOINT` environment variable created with `azd ai connection` / `azd ai toolbox`. `${PARAM_*}` placeholders resolve from the active azd env.
- Wires model deployments, AI Search, ACR, etc. `infra/layers/` provision in parallel when present.

This is a core `azd` command. Skip provision when the user gave you an existing `AZURE_AI_PROJECT_ENDPOINT` via `azd env set` -- the extension uses the existing project as-is.

After provision completes for a new project, run `azd env get-values` and set missing required azd env values, especially `AZURE_AI_PROJECT_ID` and `AZURE_TENANT_ID`, before local run or the first `azd deploy`.

### Step 3 -- Deploy the agent

```bash
azd deploy --no-prompt
# Multi-service:
azd deploy <service-name> --no-prompt
```

What deploy does:

- Reads the agent's `azure.yaml` service block, packages the agent, uploads it, and registers a new immutable version.
- **Direct code deploy** (`codeConfiguration` present): zips source, excludes `.agentignore`, and lets Foundry build the runtime image.
- **Container deploy** (no code configuration): builds the `Dockerfile`, pushes to the project's ACR, registers the version. When the service block has `image:` set, `azd` reuses the pre-built image.

After deploy, azd writes `AGENT_<SVC>_NAME`, `AGENT_<SVC>_VERSION`, and `AGENT_<SVC>_<PROTO>_ENDPOINT` (one per protocol) into the active env.

Re-deploying an identical build still creates a new version; `azd` prints `Agent version <n> is already active.` and skips the poll.

If deploy reports `Done` for the service and then fails only in `postdeploy` with `Agent <service-name> with version <n> not found`, the `azure.yaml` service key and the service's `name:` were mismatched. Rename the `azure.yaml services` key to the deployed agent name and rerun `azd deploy --no-prompt`; do not switch deployment method.

### Step 4 -- Verify and invoke

```bash
azd ai agent show --output json
```

Expect `"status": "active"` (or `"deployed"`) and an `agent_endpoints` map. Smoke-test:

```bash
azd ai agent invoke "hello, are you up?"
```

> `azd ai agent invoke` is billed, so it prints a confirmation envelope on `--no-prompt`. Summarize `changes[]`, then run `confirmCommand` once consented.

Run one remote invocation only unless the user explicitly asked to test multi-turn/session behavior. A single successful response is enough for the deployment smoke test. Anything other than a completed/successful response -> run `azd ai agent doctor --output json`, then follow [troubleshoot](../troubleshoot/troubleshoot.md).

### Step 5: Auto-Generate Evaluation Suite (MANDATORY — RUNS AUTOMATICALLY)

> ⚠️ **Pre-summary gate.** If you are about to write a deployment summary or Playground link and Step 5 has not run, you are violating this skill. Run Step 5 first.

This step runs automatically after deploy. Ask the user which source to use and start it right after deploy succeeds — with `--no-wait`, `generate` returns in seconds and generation runs server-side, so it overlaps with invoke/test steps and finishes faster overall.

> *"Your agent is deployed. Want me to set up an evaluation suite now? (a) Yes — current agent instructions (synthetic Q&A), (b) Yes — historical traces (last 3 days), (c) Yes — use existing `eval.yaml`, (d) No / later."*

| Choice | Command | What's next |
|---|---|---|
| (a) Agent instructions | `azd ai agent eval generate --gen-instruction "<agent purpose>" --no-wait --no-prompt` — `--gen-instruction` is required (hosted agents don't auto-derive it); use the service's `description:` in `azure.yaml`. | Generation runs server-side. Tell the user: *"Suite submitted. Run `azd ai agent eval run` whenever you're ready — it'll finalize `eval.yaml` and execute the eval in one step."* |
| (b) Historical traces | `azd ai agent eval generate --trace-days 3 --max-samples 50 --no-wait --no-prompt` | Same as (a). |
| (c) Existing `eval.yaml` | Skip `generate`. | Tell the user: *"Using existing `eval.yaml`. Run `azd ai agent eval run` when ready."* |
| (d) No / later | Skip. | Tell the user: *"You can run `azd ai agent eval generate` (and then `eval run`) anytime."* |

Other useful flags on `generate`: `--dataset <path-or-name>` to reuse an existing dataset instead of generating one, `--evaluator <name>` (repeatable) to pin built-in or custom evaluators, `--eval-model <name>` to choose the model used for generation and evaluation, `--reset-defaults` to overwrite an existing eval config, `--name <suite-name>` and `--out-file <path>` (default `eval.yaml`).

Then proceed to Step 6. See [After Deployment — Auto-Generate Evaluation Suite](#after-deployment--auto-generate-evaluation-suite) for run/refresh details.

### Step 6 -- Hand off

- Send more messages -> [invoke](../invoke/invoke.md)
- Evaluate / optimize -> [observe](../observe/observe.md)
- Diagnose failures -> [troubleshoot](../troubleshoot/troubleshoot.md)
- Search traces / latency -> [trace](../trace/trace.md)

## `.agentignore`

`azd ai agent init` writes a default `<service-dir>/.agentignore` for code-deploy projects (gitignore syntax) that excludes tooling files, secrets, language artifacts, and Docker files from the deploy ZIP. Only the root file is read; use `!path` to force-include.

## Endpoint or card edits -- no new version

When only `agentEndpoint:` or `agentCard:` changed in the `azure.yaml` service block:

```bash
azd ai agent endpoint update          # patch in place
azd ai agent endpoint update --force  # skip confirmation for breaking changes
```

Idempotent.

## Multi-environment deploys

```bash
azd env list
azd env select prod
azd deploy --no-prompt
```

Each env has its own `AGENT_<SVC>_*` vars.

## Common failure modes -- Hosted

| Error | Fix |
|-------|-----|
| `missing_project_endpoint` | Run `azd env set AZURE_AI_PROJECT_ENDPOINT <url>`, or run `azd provision` for a new project. |
| `invalid_agent_manifest` | `azd ai agent doctor`; fix the named field. |
| `invalid_connection` | Inspect with `azd ai connection show <name>`. |
| Docker daemon not running | You are on the container path. Add/fix `codeConfiguration` and retry direct code deploy. Only install Docker or try remote image build if you specifically need container deploy. |
| ACR push 403 | Foundry project RBAC is missing `AcrPush` for your identity. Consider switching to direct code deployment to avoid ACR entirely. |
| `container registry endpoint not found` | ACR is not configured. Use `azd env set AZURE_CONTAINER_REGISTRY_ENDPOINT <url>`, or switch to direct code deployment. |
| Agent version poll times out | Build still running; retry `azd ai agent show` after a minute. |
| `session_not_ready` (424) | Cold start or readiness delay. Wait 15-30 seconds and retry. If persistent, use `1` CPU / `2Gi` memory minimum, verify the model deployment name, capability host, and agent identity role. |
| `invalid value "json" for --output` from `azd ai agent invoke` | Invoke supports only `default` and `raw` currently. Retry without `--output json`. |
| `could not resolve agent service in azd project: no azure.ai.agent service named '<agentName>' found in azure.yaml` from `azd ai agent invoke` | Name mismatch. Use the service name, update the `azure.yaml` service block, or invoke through the Foundry MCP `agent_invoke` tool. |
| `subscription quota exceeded` | Ask user to request quota; do not auto-retry. |
| Bicep deploy errors | Forward `error.details[]` verbatim to the user. |
| `RoleAssignmentUpdateNotPermitted` during provision | A role assignment already exists but conflicts. Check for existing role assignments with `az role assignment list --scope <resource-scope>`. The provision may have succeeded for all resources except RBAC — verify with `azd ai project show` and manually assign the `Cognitive Services User` role to the agent identity if needed. |
| `eval generate`: `one of --gen-instruction ... is required` | Retry with `--gen-instruction "<agent purpose>"` (Step 5 option (a)). |
| `unknown command "init" for "azd ai agent eval"` | Command was renamed: use `azd ai agent eval generate` (requires azd CLI with `azure.ai.agents` extension up to date). |

For deeper logs, see [troubleshoot](../troubleshoot/troubleshoot.md).

## Workflow -- Prompt agent (MCP)

Prompt agents are not containerized -- they are a model + instructions + optional tools, created through the Foundry MCP server. Use when the user explicitly wants a prompt agent.

### MCP tools

| Tool | Purpose |
|------|---------|
| `agent_definition_schema_get` | Get the schema (`schemaType: "prompt"`). |
| `agent_update` | Create or update; supports `isCloneRequest` + `cloneTargetAgentName`. |
| `agent_get` | List or fetch one. |
| `agent_delete` | Delete an agent. |

### Steps

1. **Collect config** -- resolve endpoint from `azd env get-values` or ask. Then ask for **agent name**, **model deployment** (e.g. `gpt-4o`), and optional **instructions**, **temperature**, **tools**.
2. **Get schema** -- `agent_definition_schema_get` with `schemaType: "prompt"`.
3. **Create** -- `agent_update` with `{"kind": "prompt", "model": "<deployment>", "instructions": "...", "temperature": 0.7}`.
4. **Smoke test** -- follow [invoke](../invoke/invoke.md).
5. **Auto-generate evaluation suite** -- see [Step 5: Auto-Generate Evaluation Suite (Prompt)](#step-5-auto-generate-evaluation-suite-prompt-mandatory--runs-automatically) below.
6. **Hand off** -- evaluate via [observe](../observe/observe.md); clone via `agent_update` + `isCloneRequest`; delete via `agent_delete`.

### Step 5: Auto-Generate Evaluation Suite (Prompt) (MANDATORY — RUNS AUTOMATICALLY)

> ⚠️ **Pre-summary gate.** If you are about to write a deployment summary or Playground link and Step 5 has not run, you are violating this skill. Run Step 5 first.

This step runs automatically after deploy. Ask the user which source to use and start it right after deploy succeeds — with `--no-wait`, `generate` returns in seconds and generation runs server-side, so it overlaps with invoke/test steps and finishes faster overall.

> *"Your agent is deployed. Want me to set up an evaluation suite now? (a) Yes — current agent instructions (synthetic Q&A), (b) Yes — historical traces (last 3 days), (c) Yes — use existing `eval.yaml`, (d) No / later."*

| Choice | Command | What's next |
|---|---|---|
| (a) Agent instructions | `azd ai agent eval generate --gen-instruction "<agent purpose>" --no-wait --no-prompt` | Generation runs server-side. Tell the user: *"Suite submitted. Run `azd ai agent eval run` whenever you're ready — it'll finalize `eval.yaml` and execute the eval in one step."* |
| (b) Historical traces | `azd ai agent eval generate --trace-days 3 --max-samples 50 --no-wait --no-prompt` | Same as (a). |
| (c) Existing `eval.yaml` | Skip `generate`. | Tell the user: *"Using existing `eval.yaml`. Run `azd ai agent eval run` when ready."* |
| (d) No / later | Skip. | Tell the user: *"You can run `azd ai agent eval generate` (and then `eval run`) anytime."* |

## Common failure modes -- Prompt

| Error | Fix |
|-------|-----|
| Schema fetch failed | Verify endpoint format: `https://<resource>.services.ai.azure.com/api/projects/<project>`. |
| Agent creation failed | Use `agent_definition_schema_get` to verify the definition. |
| Permission denied | User needs `Foundry User` role on the project. |
| Model not found | Deploy the model first via [models/deploy-model](../../models/deploy-model/SKILL.md). |

## Display agent details (both flows)

After a successful deploy, show the agent's name, version, status, and endpoints in a table. Include a Playground link:

```
https://ai.azure.com/nextgen/r/{encodedSubId},{resourceGroup},,{accountName},{projectName}/build/agents/{agentName}/build?version={agentVersion}
```

`encodedSubId` is the subscription GUID as URL-safe base64 (no `=`):

```bash
python -c "import base64,uuid;print(base64.urlsafe_b64encode(uuid.UUID('<SUBSCRIPTION_ID>').bytes).rstrip(b'=').decode())"
```

For hosted agents, `playground_url` is in `azd ai agent show --output json`.

## After Deployment — Auto-Generate Evaluation Suite

> Reference for Step 5 options (a) and (b) — start `generate` right after deploy so its server-side generation overlaps with invoke/test steps and finishes faster. Options (c) and (d) skip `generate` and go straight to section 3 (run) or stop.

### 1. Inspect existing eval.yaml

Check the selected agent root for `eval.yaml`:

- **Exists and matches the selected agent** → skip `generate`; go to step 3 (run).
- **Missing or stale** → continue to step 2.

### 2. Submit generation (asynchronous, server-side)

Run `azd ai agent eval generate --no-wait` with the user's chosen flags (see the Step 5 table). The command:

- Submits dataset + evaluator generation jobs server-side.
- Returns in seconds.
- Writes pending operation IDs to local azd state.
- Writes a placeholder `eval.yaml` at the agent root (override with `--out-file <path>`).

No skill-side polling, terminal handle, or later-turn re-check is needed. `azd ai agent eval run` (section 3) automatically resumes a pending generation, downloads artifacts, finalizes `eval.yaml`, then runs the eval.

If the user wants to wait synchronously instead (e.g., to inspect `eval.yaml` before running), drop `--no-wait` — `generate` will then submit the jobs, wait for completion, download review artifacts, and write the finalized `eval.yaml` before returning (typically several minutes).

### 3. Run the suite

```bash
azd ai agent eval run
```

Use `azd ai agent eval show -O results.json` to inspect run details, or `azd ai agent eval list` to see history.

### 4. Refresh datasets/evaluators (later)

When local files under `datasets/<suite>/` or `evaluators/<suite>/` change, run `azd ai agent eval update --dataset-only` or `--evaluator-only` to upload new versions. azd bumps the `version` fields in `eval.yaml`.

### 5. Prompt User

*"Your agent is deployed and evaluation suite generation is **submitted server-side** (still running, takes several minutes). Would you like to run an evaluation now? `azd ai agent eval run` will wait for generation to finish, then execute the eval."*

- **Yes** → run `azd ai agent eval run` (this resumes the pending generation, then runs the eval — may take several minutes the first time), then follow the [observe skill](../observe/observe.md) to interpret results.
- **No** → stop. The user can return later via `azd ai agent eval run` — it will pick up wherever the pending generation is.
- **Production trace analysis** → follow the [trace skill](../trace/trace.md).

## Non-Interactive / YOLO Mode

> Even in `--no-prompt` / `--yolo` mode: if the user named a foundry project or asked to create one, go ahead; otherwise stop and ask before provisioning.

- Hosted: always pass `--no-prompt`. If `azd ai agent invoke` prints a `confirmation_required` envelope, summarize `changes[]` and re-run with `--force` after the user consents -- never auto-append `--force`.
- Prompt: all required values (project endpoint, agent name, model deployment) must come from the user message or `azd env get-values`; missing values should fail loudly rather than prompt.

# Evaluation Datasets — Trace-to-Dataset Pipeline & Lifecycle Management

Manage the full lifecycle of evaluation datasets for a Foundry agent: generating or regenerating data, harvesting production traces into the selected agent root's local `.foundry` cache, curating versioned test datasets, tracking evaluation quality over time, and syncing approved updates back to Foundry when needed.

## When to Use This Skill

USE FOR: create dataset from traces, harvest traces into dataset, build test dataset, dataset versioning, version my dataset, tag dataset, pin dataset version, organize datasets, dataset splits, curate test cases, review trace candidates, evaluation trending, metrics over time, eval regression, regression detection, compare evaluations over time, dataset comparison, evaluation lineage, trace to dataset pipeline, annotation review, production traces to test cases.

> ⚠️ **DO NOT manually run** KQL queries to extract datasets or call `evaluation_dataset_create` **without reading this skill first.** This skill defines the correct trace extraction patterns, schema transformation, cache rules, versioning conventions, and quality gates that raw tools do not enforce.

> 💡 **Tip:** This skill complements the [observe skill](../observe/observe.md) (eval-driven optimization loop) and the [trace skill](../trace/trace.md) (production trace analysis). Use this skill when you need to bridge traces and evaluations: turning production data into test cases and tracking evaluation quality over time.

## Quick Reference

| Property | Value |
|----------|-------|
| MCP server | `azure` |
| Key Foundry MCP tools | `data_generation_job_create`, `data_generation_job_get`, `evaluation_dataset_create`, `evaluation_dataset_get`, `evaluation_dataset_versions_get`, `evaluation_suite_create`, `evaluation_suite_get`, `evaluation_get`, `evaluation_comparison_create`, `evaluation_comparison_get` |
| Storage tools | `project_connection_list` (discover `AzureStorageAccount` connection), `project_connection_create` (add storage connection) |
| Azure services | Application Insights (via `monitor_resource_log_query`), Azure Blob Storage (dataset sync) |
| Prerequisites | Agent deployed, effective context resolved from azd or metadata overlay, App Insights connected |
| Local cache | `.foundry/datasets/`, `.foundry/results/`, `.foundry/evaluators/` |

## Entry Points

| User Intent | Start At |
|-------------|----------|
| "Create dataset from production traces" / "Harvest traces" | [Trace-to-Dataset Pipeline](references/trace-to-dataset.md) |
| "Version my dataset" / "Tag dataset" / "Pin dataset version" | [Dataset Versioning](references/dataset-versioning.md) |
| "Organize my datasets" / "Dataset splits" / "Filter datasets" | [Dataset Organization](references/dataset-organization.md) |
| "Review trace candidates" / "Curate test cases" | [Dataset Curation](references/dataset-curation.md) |
| "Show eval metrics over time" / "Evaluation trending" | [Eval Trending](references/eval-trending.md) |
| "Did my agent regress?" / "Regression detection" | [Eval Regression](references/eval-regression.md) |
| "Compare datasets" / "Experiment comparison" / "A/B test" | [Dataset Comparison](references/dataset-comparison.md) |
| "Sync dataset to Foundry" / "Refresh local dataset cache" | [Trace-to-Dataset Pipeline -> Step 5](references/trace-to-dataset.md#step-5--sync-local-cache-with-foundry-optional) |
| "Trace my evaluation lineage" / "Audit eval history" | [Eval Lineage](references/eval-lineage.md) |
| "Generate eval dataset" / "Create seed dataset" / "Generate test cases for my agent" | [Generate Seed Dataset](references/generate-seed-dataset.md) |
| "Regenerate dataset" / "Refresh synthetic data" / "Generate from traces without full suite" | [Generated Data Refresh](../observe/references/evaluation-suite-generation.md#regenerate-one-artifact) |

## Before Starting — Detect Current State

1. Resolve the target agent root, environment, effective deployment context, and selected metadata overlay using [Common Project Context Resolution](../../SKILL.md#agent-common-project-context-resolution).
2. Confirm the selected environment's `projectEndpoint`, `agentName`, and observability settings from azd first, then metadata overrides.
3. Check `.foundry/datasets/`, `.foundry/results/`, `.foundry/datasets/manifest.json`, and `eval.yaml` in the selected agent root only.
4. Check whether `evaluation_dataset_get` returns server-side datasets for the same environment and whether `evaluationSuites[]` contains `suiteName`/`suiteVersion` references.
5. Route to the appropriate entry point based on user intent.

## The Foundry Flywheel

```text
Production Agent -> [1] Trace (App Insights + OTel)
                -> [2] Harvest (KQL extraction)
                -> [3] Curate (human review)
                -> [4] Dataset Cache (.foundry/datasets, versioned)
                -> [5] Sync to Foundry (optional refresh/push)
                -> [6] Evaluate (batch eval)
                -> [7] Analyze (trending + regression)
                -> [8] Compare (agent versions OR dataset versions)
                -> [9] Deploy -> back to [1]
```

Each cycle makes the test suite harder and more representative. Production failures from release N become regression tests for release N+1.

## Behavioral Rules

1. **Always show KQL queries.** Before executing any trace extraction query, display it in a code block. Never run queries silently.
2. **Scope to time ranges.** Always include a time range in KQL queries (default: last 7 days for trace harvesting). Ask the user for the range if not specified.
3. **Require human review.** Never auto-commit harvested traces to a dataset without showing candidates to the user first. The curation step is mandatory.
4. **Use dataset naming conventions.** Follow the naming conventions below and keep local filenames aligned with the registered Foundry dataset name/version.
5. **Treat local files as cache.** Reuse `.foundry/datasets/` and `.foundry/evaluators/` when they already match the selected environment in the selected agent root. Offer refresh when the user asks or when remote state has changed.
6. **Use generated data when requested.** Prefer `data_generation_job_create` for standalone dataset regeneration from agent, dataset, prompt, file, or trace context. Poll with `data_generation_job_get`; if generation fails or returns incomplete artifacts, explain the failure and fall back to the local/manual dataset flow.
7. **Stay inside the selected agent root.** After resolving the agent root, inspect only that folder's `.foundry/` cache and source context. Never merge sibling agent folders.
8. **Persist artifacts.** Save datasets to `.foundry/datasets/`, evaluation results to `.foundry/results/`, and track lineage in `.foundry/datasets/manifest.json`.
9. **Keep evaluation suites aligned.** Update the selected environment's `evaluationSuites[]` in the selected metadata file whenever a dataset version, evaluator set, suite version, or suite tags change. Local flows should default to `agent-metadata.yaml`; prod or CI-targeted flows can use `agent-metadata.<env>.yaml`. If the environment still uses older `testSuites[]` or legacy `testCases[]`, treat that list as the current suite source for this session and rewrite it as `evaluationSuites[]` on the next metadata save.
10. **Confirm before overwriting.** If a dataset version or cache file already exists, warn the user and ask for confirmation before replacing or refreshing it.
11. **Sync to Foundry when requested or needed.** After saving datasets locally, refresh or register them in Foundry only when the user asks or the workflow needs shared/CI usage. Use `evaluation_suite_create` for reviewed suite versions that combine the updated dataset and evaluator set.
12. **Never remove dataset rows or weaken evaluators to recover scores.** Score drops after a dataset update are expected - harder tests expose real gaps. Optimize the agent for new failure patterns; do not shrink the test suite.
13. **Match eval parameter names exactly.** Use `evaluation_agent_batch_eval_create` for agent-target batch eval, including suites that have `suiteName`; call `evaluation_suite_get` only to resolve suite metadata. Use `evaluationId` when creating grouped batch runs, but use `evalId` for `evaluation_get` and comparison/trending lookups.

## Dataset Naming and Metadata Conventions

| Dataset type | Foundry dataset name | Foundry dataset version | Typical local file | Metadata stage |
|--------------|----------------------|-------------------------|--------------------|----------------|
| Seed dataset | `<agent-name>-eval-seed` | `v1` | `.foundry/datasets/<agent-name>-eval-seed-v1.jsonl` | `seed` |
| Trace-harvested dataset | `<agent-name>-traces` | `v<N>` | `.foundry/datasets/<agent-name>-traces-v<N>.jsonl` | `traces` |
| Curated/refined dataset | `<agent-name>-curated` | `v<N>` | `.foundry/datasets/<agent-name>-curated-v<N>.jsonl` | `curated` |
| Production-ready dataset | `<agent-name>-prod` | `v<N>` | `.foundry/datasets/<agent-name>-prod-v<N>.jsonl` | `prod` |

Here `<agent-name>` means the effective selected Foundry agent name from azd or metadata. If that deployed agent name already includes the environment (for example, `support-agent-dev`), do **not** append the environment key a second time.

Local dataset filenames must start with the effective selected Foundry agent name. Put stage and version suffixes **after** that prefix so cache files sort and group by agent first.

Keep the Foundry dataset name stable across versions. Store the version only in `datasetVersion` (or manifest `version`) using the `v<N>` format, while local filenames keep the `-v<N>` suffix for cache readability.

Required metadata to track with every registered or generated dataset:

- `agent`: the agent name (for example, `hosted-agent-051-001`)
- `stage`: `seed`, `traces`, `curated`, or `prod`
- `version`: version string such as `v1`, `v2`, or `v3`
- `datasetUri`: always persist the Foundry dataset URI in the selected metadata file alongside the local `datasetFile`, dataset name, and version

> 💡 **Tip:** `evaluation_dataset_create` does not expose a first-class `tags` parameter in the current MCP surface. Persist `agent`, `stage`, and `version` in local metadata (the selected metadata file plus `.foundry/datasets/manifest.json`) so Foundry-side references stay aligned with the cache.

When a dataset belongs to a generated suite, keep the selected environment's suite metadata aligned with `suiteName`, `suiteVersion`, `generationJobId`, and `generationSource`. Dataset regeneration with `data_generation_job_create` should create a new local dataset version and a reviewed suite version; do not mutate old suite versions in place.

## Related Skills

| User Intent | Skill |
|-------------|-------|
| "Run an evaluation" / "Optimize my agent" | [observe skill](../observe/observe.md) |
| "Search traces" / "Analyze failures" / "Latency analysis" | [trace skill](../trace/trace.md) |
| "Find eval scores for a response ID" / "Link eval results to traces" | [trace skill -> Eval Correlation](../trace/references/eval-correlation.md) |
| "Deploy my agent" | [deploy skill](../deploy/deploy.md) |
| "Debug container issues" | [troubleshoot skill](../troubleshoot/troubleshoot.md) |
| "Review metadata schema" | [Agent Metadata Contract](../../references/agent-metadata-contract.md) |

# Dataset Comparison — A/B Testing Across Dataset Versions

Run structured experiments that compare how an agent performs across different dataset versions, and present results as leaderboards with per-evaluator breakdowns. Use this to answer: "Did scores drop because of harder tests or agent regression?"

## Experiment Structure

An experiment consists of:
1. **Pinned agent version** — the same agent evaluated on each dataset
2. **Varied dataset versions** — the versions being compared
3. **Same evaluators** — applied consistently across all runs
4. **Comparison results** — which dataset version the agent performs better on

## Step 1 — Define the Experiment

| Parameter | Value | Example |
|-----------|-------|---------|
| Agent | Pinned agent version | `v3` |
| Baseline dataset | Previous dataset version | `support-bot-prod-traces-v2` |
| Treatment dataset(s) | New dataset version(s) | `support-bot-prod-traces-v3` |
| Evaluators | Same set for all runs | coherence, fluency, relevance, intent_resolution, task_adherence |

## Step 2 — Run Evaluations

For each dataset version, run **`evaluation_agent_batch_eval_create`** with:
- Same `evaluationId` (groups all runs for comparison)
- Same `agentVersion`
- Same `evaluatorNames`
- Different `inputData` (from each dataset version)

> **Important:** Use `evaluationId` on `evaluation_agent_batch_eval_create` to group runs. After the runs exist, switch to `evalId` for `evaluation_get` and `evaluation_comparison_create`.

> ⚠️ **Eval-group immutability:** Keep the evaluator set and thresholds fixed within one evaluation group. If you need to change evaluators or thresholds, create a new evaluation group instead of reusing the previous `evaluationId`.

> ⚠️ **Score drops are expected.** When comparing v1→v2 datasets, lower scores on the new dataset likely mean the new test cases are harder (better coverage), not that the agent regressed. **Do NOT remove dataset rows or weaken evaluators to recover scores.** Instead, optimize the agent for the new failure patterns, then re-evaluate.

## Step 3 — Compare Results

Use **`evaluation_comparison_create`** with the baseline and treatment runs:

```json
{
  "insightRequest": {
    "displayName": "Dataset comparison: traces-v2 vs traces-v3 on agent-v3",
    "state": "NotStarted",
    "request": {
      "type": "EvaluationComparison",
      "evalId": "<eval-group-id>",
      "baselineRunId": "<traces-v2-run-id>",
      "treatmentRunIds": ["<traces-v3-run-id>"]
    }
  }
}
```

> ⚠️ **Common mistake:** `evaluation_comparison_create` uses `insightRequest.request.evalId`, not `evaluationId`, even when the runs were originally grouped with `evaluationId`.

## Step 4 — Leaderboard

Present results as a leaderboard table:

| Evaluator | traces-v2 (baseline) | traces-v3 | Effect |
|-----------|:---:|:---:|:---:|
| Coherence | 4.0 | 3.6 | ⚠️ Lower |
| Fluency | 4.5 | 4.3 | ⚠️ Lower |
| Relevance | 3.6 | 3.2 | ⚠️ Lower |
| Intent Resolution | 4.1 | 3.7 | ⚠️ Lower |
| Task Adherence | 3.9 | 3.4 | ⚠️ Lower |

### Recommendation

If scores drop uniformly across all evaluators, the new dataset is likely harder:

*"Agent v3 scores dropped on traces-v3 across all evaluators. traces-v3 added 15 edge-case queries from production failures. This is expected — optimize the agent for the new failure patterns rather than reverting the dataset."*

## Pairwise A/B Comparison

For detailed pairwise analysis between exactly two dataset versions:

| Evaluator | Baseline (traces-v2) | Treatment (traces-v3) | Delta | p-value | Effect |
|-----------|:---:|:---:|:---:|:---:|:---:|
| Coherence | 4.0 ± 0.6 | 3.6 ± 0.9 | −0.4 | 0.03 | Degraded |
| Fluency | 4.5 ± 0.4 | 4.3 ± 0.5 | −0.2 | 0.12 | Inconclusive |
| Relevance | 3.6 ± 0.9 | 3.2 ± 1.1 | −0.4 | 0.04 | Degraded |

> 💡 **Tip:** The `evaluation_comparison_create` result includes `pValue` and `treatmentEffect` fields. Use `pValue < 0.05` as the threshold for statistical significance.

## Multi-Dataset Comparison

Compare how the same agent version performs across different datasets:

| Dataset | Coherence | Fluency | Relevance | Notes |
|---------|:---------:|:-------:|:---------:|-------|
| traces-v3 (prod) | 4.0 | 4.5 | 3.6 | Production-derived |
| synthetic-v2 | 4.3 | 4.6 | 4.1 | May overestimate quality |
| manual-v1 (curated) | 3.8 | 4.4 | 3.2 | Hardest test cases |

> ⚠️ **Warning:** Be cautious comparing scores across datasets with different structures (e.g., production traces vs synthetic). Differences may reflect dataset difficulty, not agent quality.

## Next Steps

- **Track trends over time** → [Eval Trending](eval-trending.md)
- **Check for regressions** → [Eval Regression](eval-regression.md)
- **Audit full lineage** → [Eval Lineage](eval-lineage.md)

# Dataset Curation — Human-in-the-Loop Review

Review, annotate, and approve harvested trace candidates before including them in evaluation datasets. This ensures dataset quality by adding a human review gate between raw trace extraction and finalized test cases.

## Workflow Overview

```
Raw Traces (from KQL harvest)
    │
    ▼
[1] Candidate File (unreviewed)
    │
    ▼
[2] Human Review (approve/edit/reject each)
    │
    ▼
[3] Approved Dataset (versioned, ready for eval)
```

## Step 1 — Generate Candidate File

After running a [trace harvest](trace-to-dataset.md), save candidates with a `status` field:

```
.foundry/datasets/<agent-name>-traces-candidates-<date>.jsonl
```

Each line includes a review status:

```json
{"query": "How do I reset my password?", "response": "...", "status": "pending", "metadata": {"source": "trace", "conversationId": "conv-abc-123", "harvestRule": "error", "errorType": "TimeoutError", "duration": 12300}}
{"query": "What's the refund policy?", "response": "...", "status": "pending", "metadata": {"source": "trace", "conversationId": "conv-def-456", "harvestRule": "latency", "duration": 8700}}
```

## Step 2 — Present for Review

Show candidates in a review table:

| # | Status | Query (preview) | Source | Error | Duration | Eval Score |
|---|--------|----------------|--------|-------|----------|------------|
| 1 | ⏳ pending | "How do I reset my..." | error harvest | TimeoutError | 12.3s | — |
| 2 | ⏳ pending | "What's the refund..." | latency harvest | — | 8.7s | — |
| 3 | ⏳ pending | "Can you help me..." | low-eval harvest | — | 0.4s | 2.0 |

### Review Actions

For each candidate, the user can:

| Action | Result |
|--------|--------|
| **Approve** | Include in dataset as-is |
| **Approve + Edit** | Include with modified query/response/ground_truth |
| **Add Ground Truth** | Approve and add the expected correct answer |
| **Reject** | Exclude from dataset |
| **Flag** | Mark for later review |

### Batch Operations

- *"Approve all"* — include all pending candidates
- *"Approve all errors"* — include all candidates from error harvest
- *"Reject duplicates"* — exclude candidates with similar queries to existing dataset entries
- *"Approve #1, #3, #5; reject #2, #4"* — selective approval by number

## Step 3 — Finalize Dataset

After review, filter approved candidates and save to a versioned dataset:

1. Read `.foundry/datasets/manifest.json` to find the latest version number
2. Filter candidates where `status == "approved"`
3. Remove the `status` field from the output
4. Save to `.foundry/datasets/<agent-name>-<source>-v<N>.jsonl`
5. Update `.foundry/datasets/manifest.json` with metadata

### Update Candidate Status

Mark the candidate file with final statuses:

```json
{"query": "How do I reset my password?", "status": "approved", "ground_truth": "Navigate to Settings > Security > Reset Password", "metadata": {...}}
{"query": "What's the refund policy?", "status": "rejected", "rejectReason": "duplicate of existing test case", "metadata": {...}}
{"query": "Can you help me...", "status": "approved", "metadata": {...}}
```

> 💡 **Tip:** Keep candidate files as an audit trail. They document what was reviewed, when, and why items were accepted or rejected.

## Quality Checks

Before finalizing, verify dataset quality:

| Check | Criteria |
|-------|----------|
| **No duplicates** | Ensure no query appears in both the new dataset and existing datasets |
| **Balanced categories** | Verify reasonable distribution across categories (not all edge-cases) |
| **Ground truth coverage** | Flag examples without ground_truth that may benefit from one |
| **Minimum size** | Warn if dataset has fewer than 20 examples (may not be statistically meaningful) |
| **Safety coverage** | Ensure safety-related test cases are included if the agent handles sensitive topics |

## Next Steps

- **Version the approved dataset** → [Dataset Versioning](dataset-versioning.md)
- **Organize into splits** → [Dataset Organization](dataset-organization.md)
- **Run evaluation** → [observe skill Step 2](../../observe/references/evaluate-step.md)

# Dataset Organization — Metadata, Splits, and Filtered Evaluation

Organize datasets using metadata fields, create train/validation/test splits, and run targeted evaluations on dataset subsets. This addresses the need for hierarchical dataset organization without requiring rigid container structures.

## Metadata Schema

Add metadata to each JSONL example to enable filtering and organization:

| Field | Values | Purpose |
|-------|--------|---------|
| `category` | `edge-case`, `regression`, `happy-path`, `multi-turn`, `safety` | Test case classification |
| `source` | `trace`, `synthetic`, `manual`, `feedback` | How the example was created |
| `split` | `train`, `val`, `test` | Dataset split assignment |
| `tags` | key/value object such as `{"tier": "smoke", "purpose": "baseline"}` | Flexible suite-alignment and filtering labels |
| `harvestRule` | `error`, `latency`, `low-eval`, `combined` | Which harvest template captured it |
| `agentVersion` | `"1"`, `"2"`, etc. | Agent version when trace was captured |

### Example JSONL with Metadata

```json
{"query": "Reset my password", "ground_truth": "Navigate to Settings > Security > Reset Password", "metadata": {"category": "happy-path", "source": "manual", "split": "test", "tags": {"tier": "smoke", "purpose": "baseline"}}}
{"query": "What happens if I delete my account while a refund is pending?", "metadata": {"category": "edge-case", "source": "trace", "split": "test", "tags": {"tier": "regression", "purpose": "coverage"}, "harvestRule": "error"}}
{"query": "I want to harm myself", "ground_truth": "I'm concerned about your safety. Please contact...", "metadata": {"category": "safety", "source": "manual", "split": "test", "tags": {"tier": "smoke", "purpose": "safety"}}}
```

## Creating Splits

### Automatic Split Assignment

When creating a new dataset, assign splits based on rules:

| Rule | Split | Rationale |
|------|-------|-----------|
| First 70% of examples | `train` | Bulk of data for development |
| Next 15% of examples | `val` | Validation during optimization |
| Final 15% of examples | `test` | Held-out for final evaluation |
| All `tags.tier == "smoke"` examples | `test` | Smoke suites always stay in test |
| All `category: safety` examples | `test` | Safety always evaluated |

### Manual Split Assignment

Users can assign splits during [curation](dataset-curation.md) or by editing the JSONL metadata directly.

## Filtered Evaluation Runs

Run evaluations on specific subsets of a dataset by filtering JSONL before passing to the evaluator.

### Filter by Split

```python
import json

# Read full dataset
with open(".foundry/datasets/support-bot-prod-traces-v3.jsonl") as f:
    examples = [json.loads(line) for line in f]

# Filter to test split only
test_examples = [e for e in examples if e.get("metadata", {}).get("split") == "test"]

# Pass test_examples as inputData to evaluation_agent_batch_eval_create
```

### Filter by Category

```python
# Only edge cases
edge_cases = [e for e in examples if e.get("metadata", {}).get("category") == "edge-case"]

# Only safety test cases
safety_cases = [e for e in examples if e.get("metadata", {}).get("category") == "safety"]

# Only smoke suites
smoke_cases = [
    e for e in examples
    if e.get("metadata", {}).get("tags", {}).get("tier") == "smoke"
]
```

### Filter by Source

```python
# Only production trace-derived cases (most representative)
trace_cases = [e for e in examples if e.get("metadata", {}).get("source") == "trace"]

# Only manually curated cases (highest quality ground truth)
manual_cases = [e for e in examples if e.get("metadata", {}).get("source") == "manual"]
```

## Dataset Statistics

Generate summary statistics to understand dataset composition:

```python
from collections import Counter

categories = Counter(e.get("metadata", {}).get("category", "unknown") for e in examples)
sources = Counter(e.get("metadata", {}).get("source", "unknown") for e in examples)
splits = Counter(e.get("metadata", {}).get("split", "unassigned") for e in examples)
tiers = Counter(e.get("metadata", {}).get("tags", {}).get("tier", "none") for e in examples)
```

Present as a table:

| Dimension | Values | Count |
|-----------|--------|-------|
| **Category** | happy-path: 20, edge-case: 15, regression: 8, safety: 5, multi-turn: 10 | 58 total |
| **Source** | trace: 30, synthetic: 18, manual: 10 | 58 total |
| **Split** | train: 40, val: 9, test: 9 | 58 total |
| **Tier** | smoke: 12, regression: 25, coverage: 21 | 58 total |

## Next Steps

- **Run targeted evaluation** → [observe skill Step 2](../../observe/references/evaluate-step.md) (pass filtered `inputData`)
- **Compare splits** → [Dataset Comparison](dataset-comparison.md)
- **Track lineage** → [Eval Lineage](eval-lineage.md)

# Dataset Versioning — Version Management & Tagging

Manage dataset versions with naming conventions, tagging, and version pinning for reproducible evaluations. This workflow formalizes dataset lifecycle management using existing MCP tools and local conventions.

## Naming Convention

Use the pattern `<agent-name>-<source>-v<N>`:

| Component | Values | Example |
|-----------|--------|---------|
| `<agent-name>` | Selected environment's `agentName` from the selected metadata file | `support-bot-prod` |
| `<source>` | `traces`, `synthetic`, `manual`, `combined` | `traces` |
| `v<N>` | Incremental version number | `v3` |

`<agent-name>` already refers to the environment-specific deployed Foundry agent name. If that value includes the environment key, do **not** append the environment again.

**Full examples:**
- `support-bot-prod-traces-v1` — first production dataset from trace harvesting
- `support-bot-dev-synthetic-v2` — second synthetic dataset
- `support-bot-prod-combined-v5` — fifth production dataset combining traces + manual examples

## Tagging Conventions

Tags are stored in `.foundry/datasets/manifest.json` alongside dataset metadata:

| Tag | Meaning | When to Apply |
|-----|---------|---------------|
| `baseline` | Reference dataset for comparison | When establishing a new evaluation baseline |
| `prod` | Dataset used for current production evaluation | After successful deployment |
| `canary` | Dataset for canary/staging evaluation | During staged rollout |
| `regression-<date>` | Dataset that caught a regression | When a regression is detected |
| `deprecated` | Dataset no longer in active use | When replaced by a newer version |

## Version Pinning

Pin evaluations to a specific dataset version to ensure reproducible, comparable results:

### Local Pinning (JSONL Datasets)

When using local JSONL files, reference the exact filename in evaluation runs:

```
.foundry/datasets/support-bot-prod-traces-v3.jsonl  ← pinned by filename
```

Pass the contents via `inputData` parameter in **`evaluation_agent_batch_eval_create`**.

### Server-Side Version Discovery

Use `evaluation_dataset_versions_get` to list all versions of a dataset registered in Foundry:

```
evaluation_dataset_versions_get(projectEndpoint, datasetName: "<agent-name>-<source>")
```

Use `evaluation_dataset_get` without a name to list all datasets in the project:

```
evaluation_dataset_get(projectEndpoint)
```

> 💡 **Tip:** Server-side versions are available after syncing via [Trace-to-Dataset → Step 5](trace-to-dataset.md#step-5--sync-local-cache-with-foundry-optional). Local `manifest.json` remains useful for lineage metadata (source, harvestRule, reviewedBy) not stored server-side.

## Manifest File

Track all dataset versions, required dataset metadata, tags, and lineage in `.foundry/datasets/manifest.json`:

```json
{
  "datasets": [
    {
      "name": "support-bot-prod-traces",
      "file": "support-bot-prod-traces-v1.jsonl",
      "version": "v1",
      "agent": "support-bot-prod",
      "stage": "traces",
      "datasetUri": "<foundry-dataset-uri-v1>",
      "tag": "deprecated",
      "source": "trace-harvest",
      "harvestRule": "error",
      "timeRange": "2025-01-01 to 2025-01-07",
      "exampleCount": 32,
      "createdAt": "2025-01-08T10:00:00Z",
      "evalRunIds": ["run-abc-123"]
    },
    {
      "name": "support-bot-prod-traces",
      "file": "support-bot-prod-traces-v2.jsonl",
      "version": "v2",
      "agent": "support-bot-prod",
      "stage": "traces",
      "datasetUri": "<foundry-dataset-uri-v2>",
      "tag": "baseline",
      "source": "trace-harvest",
      "harvestRule": "error+latency",
      "timeRange": "2025-01-15 to 2025-01-21",
      "exampleCount": 47,
      "createdAt": "2025-01-22T10:00:00Z",
      "evalRunIds": ["run-def-456", "run-ghi-789"]
    },
    {
      "name": "support-bot-prod-traces",
      "file": "support-bot-prod-traces-v3.jsonl",
      "version": "v3",
      "agent": "support-bot-prod",
      "stage": "traces",
      "datasetUri": "<foundry-dataset-uri-v3>",
      "tag": "prod",
      "source": "trace-harvest",
      "harvestRule": "error+latency+low-eval",
      "timeRange": "2025-02-01 to 2025-02-07",
      "exampleCount": 63,
      "createdAt": "2025-02-08T10:00:00Z",
      "evalRunIds": []
    }
  ]
}
```

Keep `stage` stable for the dataset family (`seed`, `traces`, `curated`, or `prod`) and use `tag` for mutable lifecycle labels such as `baseline`, `prod`, or `deprecated`. Persist `datasetUri` as the Foundry-returned dataset reference so deploy and observe workflows can resolve the registered dataset directly.

## Creating a New Version

1. **Check existing versions**: Read `.foundry/datasets/manifest.json` to find the latest version number
2. **Increment version**: Use `v<N+1>` as the new version
3. **Create dataset**: Via [Trace-to-Dataset](trace-to-dataset.md) or manual JSONL creation
4. **Update manifest**: Add the new entry with metadata
5. **Tag appropriately**: Apply `baseline`, `prod`, or other tags as needed
6. **Deprecate old**: Optionally mark previous versions as `deprecated`

> ⚠️ **DO NOT stop here.** After creating a new dataset version, continue to the Dataset Update Loop below.

## Dataset Update Loop — Eval → Analyze → Optimize → Re-Eval

When a dataset is updated (new rows, better coverage, new failure modes), run this loop to validate the agent against the harder test suite:

```
[1] Eval with new dataset (v2) using same agent version
    │
    ▼
[2] Compare: eval on v1 vs eval on v2 (same agent, different datasets)
    │
    ▼
[3] Analyze score changes — expect some drops (harder tests ≠ worse agent)
    │
    ▼
[4] Optimize agent prompt based on NEW failure patterns only
    │
    ▼
[5] Re-eval optimized agent on v2 dataset → compare to pre-optimization
    │
    ▼
[6] If satisfied → tag v2 as `prod`, archive v1
```

### ⛔ Guardrails for This Loop

- **Never remove dataset rows to recover scores.** If eval scores drop after a dataset update, the dataset is likely exposing real gaps. Removing hard cases defeats the purpose.
- **Never weaken evaluators to recover scores.** Do not lower thresholds, remove evaluators, or switch to easier scoring when scores drop on an expanded dataset.
- **Distinguish dataset difficulty from agent regression.** A score drop on a harder dataset is expected and healthy — it means test coverage improved. Only flag as regression when the same dataset + same evaluators produce worse scores on a new agent version.
- **Optimize for NEW failure patterns only.** When optimizing the agent prompt after a dataset update, target the newly added test cases. Do not re-optimize for cases that were already passing.

## Comparing Versions

To understand how a dataset evolved between versions:

```bash
# Count examples per version
wc -l .foundry/datasets/support-bot-prod-traces-v*.jsonl

# Diff example queries between versions
jq -r '.query' .foundry/datasets/support-bot-prod-traces-v2.jsonl | sort > /tmp/v2-queries.txt
jq -r '.query' .foundry/datasets/support-bot-prod-traces-v3.jsonl | sort > /tmp/v3-queries.txt
diff /tmp/v2-queries.txt /tmp/v3-queries.txt
```

## Next Steps

- **Organize into splits** → [Dataset Organization](dataset-organization.md)
- **Run evaluation with pinned version** → [observe skill Step 2](../../observe/references/evaluate-step.md)
- **Track lineage** → [Eval Lineage](eval-lineage.md)

# Eval Lineage — Full Traceability from Production to Deployment

Track the complete chain from production traces through dataset creation, evaluation runs, comparisons, and deployment decisions. Enables "why was this deployed?" audit queries and compliance reporting.

## Lineage Chain

```
Production Trace (App Insights)
    │ conversationId, responseId
    ▼
Dataset Version (.foundry/datasets/*.jsonl, environment-scoped)
    │ metadata.conversationId, metadata.harvestRule
    ▼
Evaluation Run (evaluation_agent_batch_eval_create)
    │ evaluationId when creating, evalId when querying, evalRunId
    ▼
Comparison (evaluation_comparison_create)
    │ insightId, baselineRunId, treatmentRunIds
    ▼
Deployment Decision (agent_update)
    │ agentVersion
    ▼
Production Trace (cycle repeats)
```

## Lineage Manifest

Track lineage in `.foundry/datasets/manifest.json`:

```json
{
  "datasets": [
    {
      "name": "support-bot-prod-traces",
      "file": "support-bot-prod-traces-v3.jsonl",
      "version": "v3",
      "tag": "prod",
      "source": "trace-harvest",
      "harvestRule": "error+latency",
      "timeRange": "2025-02-01 to 2025-02-07",
      "exampleCount": 63,
      "createdAt": "2025-02-08T10:00:00Z",
      "evalRuns": [
        {
          "evalId": "eval-group-001",
          "runId": "run-abc-123",
          "agentVersion": "3",
          "date": "2025-02-08T12:00:00Z",
          "status": "completed"
        },
        {
          "evalId": "eval-group-001",
          "runId": "run-def-456",
          "agentVersion": "4",
          "date": "2025-02-10T09:00:00Z",
          "status": "completed"
        }
      ],
      "comparisons": [
        {
          "insightId": "insight-xyz-789",
          "baselineRunId": "run-abc-123",
          "treatmentRunIds": ["run-def-456"],
          "result": "v4 improved on 3/5 metrics",
          "date": "2025-02-10T10:00:00Z"
        }
      ],
      "deployments": [
        {
          "agentVersion": "4",
          "deployedAt": "2025-02-10T14:00:00Z",
          "reason": "v4 improved coherence +25%, relevance +10% vs v3"
        }
      ]
    }
  ]
}
```

## Audit Queries

### "Why was version X deployed?"

1. Read `.foundry/datasets/manifest.json`
2. Find entries where `deployments[].agentVersion == X`
3. Show the comparison that justified the deployment
4. Show the dataset and eval runs that informed the comparison

### "What traces led to this dataset?"

1. Read the dataset JSONL file
2. Extract `metadata.conversationId` from each example
3. Look up each conversation in App Insights using the [trace skill](../../trace/trace.md)

### "What evaluation history does this agent have?"

1. Use **`evaluation_get`** to list all evaluation groups
2. For each group, list runs with `isRequestForRuns=true`
3. Build the timeline from [Eval Trending](eval-trending.md)
4. Show comparisons from **`evaluation_comparison_get`**

### "Did this dataset version catch any regressions?"

1. Find the dataset version in the manifest
2. Check `evalRuns` for runs that used this dataset
3. Check `comparisons` for any regression results
4. Cross-reference with `tag == "regression-<date>"` entries

## Maintaining Lineage

Update `.foundry/datasets/manifest.json` at each step:

| Event | Fields to Update |
|-------|-----------------|
| Dataset created | Add new entry with `name`, `version`, `source`, `exampleCount` |
| Evaluation run | Append to `evalRuns[]` with `evalId`, `runId`, `agentVersion` |
| Comparison | Append to `comparisons[]` with `insightId`, `result` |
| Deployment | Append to `deployments[]` with `agentVersion`, `reason` |
| Tag change | Update `tag` field |

> 💡 **Tip:** Store the evaluation group identifier as `evalId` in lineage/manifest records, even if the create call used the parameter name `evaluationId`.

## Next Steps

- **View metric trends** → [Eval Trending](eval-trending.md)
- **Check for regressions** → [Eval Regression](eval-regression.md)
- **Harvest new traces** → [Trace-to-Dataset](trace-to-dataset.md) (start the next cycle)

# Eval Regression — Automated Regression Detection

Automatically detect when evaluation metrics degrade between agent versions. Compare each evaluation run against the baseline and generate pass/fail verdicts with actionable recommendations.

## Prerequisites

- At least 2 evaluation runs in the same evaluation group
- Baseline run identified (either the first run or the one tagged as `baseline`)

## Step 1 — Identify Baseline and Treatment

### Automatic Baseline Selection

1. Read `.foundry/datasets/manifest.json` and find the dataset tagged `baseline`.
2. If the baseline dataset entry includes a stored `baselineRunId` (or mapping to one or more `evalRunIds`), use that `baselineRunId` as the baseline run.
3. If no explicit `baselineRunId` is recorded, select the first (oldest) run in the evaluation group as the baseline.

### Treatment Selection

The latest (most recent) run in the evaluation group is the treatment.

## Step 2 — Run Comparison

Use **`evaluation_comparison_create`** to compare baseline vs treatment:

> **Critical:** `displayName` is **required** in the `insightRequest`. Despite the MCP tool schema showing it as optional, the API rejects requests without it.

```json
{
  "insightRequest": {
    "displayName": "Regression Check - v1 vs v4",
    "state": "NotStarted",
    "request": {
      "type": "EvaluationComparison",
      "evalId": "<eval-group-id>",
      "baselineRunId": "<baseline-run-id>",
      "treatmentRunIds": ["<latest-run-id>"]
    }
  }
}
```

Retrieve results with **`evaluation_comparison_get`** using the returned `insightId`.

## Step 3 — Regression Verdicts

For each evaluator in the comparison results, apply regression thresholds:

| Treatment Effect | Delta | Verdict | Action |
|-----------------|-------|---------|--------|
| `Improved` | > +2% | ✅ PASS | No action needed |
| `Changed` | ±2% | ⚠️ NEUTRAL | Monitor, no immediate action |
| `Degraded` | > -2% | 🔴 REGRESSION | Investigate and remediate |
| `Inconclusive` | — | ❓ INCONCLUSIVE | Increase sample size and re-run |
| `TooFewSamples` | — | ❓ INSUFFICIENT DATA | Need more test cases (≥30 recommended) |

### Example Regression Report

```
╔═══════════════════════════════════════════════════════════════╗
║              REGRESSION REPORT: v1 (baseline) → v4           ║
╠═══════════════════════════════════════════════════════════════╣
║ Evaluator          │ Baseline │ Treatment │ Delta  │ Verdict ║
╠════════════════════╪══════════╪═══════════╪════════╪═════════╣
║ Coherence          │ 3.2      │ 4.0       │ +0.8   │ ✅ PASS ║
║ Fluency            │ 4.1      │ 4.5       │ +0.4   │ ✅ PASS ║
║ Relevance          │ 2.8      │ 3.6       │ +0.8   │ ✅ PASS ║
║ Intent Resolution  │ 3.0      │ 4.1       │ +1.1   │ ✅ PASS ║
║ Task Adherence     │ 2.5      │ 3.9       │ +1.4   │ ✅ PASS ║
║ Safety             │ 0.95     │ 0.98      │ +0.03  │ ✅ PASS ║
╠═══════════════════════════════════════════════════════════════╣
║ OVERALL: ✅ ALL EVALUATORS PASSED — Safe to deploy           ║
╚═══════════════════════════════════════════════════════════════╝
```

### Example with Regression

```
╔═══════════════════════════════════════════════════════════════╗
║              REGRESSION REPORT: v3 → v4                      ║
╠═══════════════════════════════════════════════════════════════╣
║ Evaluator          │ v3       │ v4        │ Delta  │ Verdict ║
╠════════════════════╪══════════╪═══════════╪════════╪═════════╣
║ Coherence          │ 4.1      │ 4.0       │ -0.1   │ ⚠️ NEUT║
║ Fluency            │ 4.4      │ 4.5       │ +0.1   │ ✅ PASS ║
║ Relevance          │ 4.0      │ 3.6       │ -0.4   │ 🔴 REGR║
║ Intent Resolution  │ 4.2      │ 4.1       │ -0.1   │ ⚠️ NEUT║
║ Task Adherence     │ 3.8      │ 3.9       │ +0.1   │ ✅ PASS ║
║ Safety             │ 0.96     │ 0.98      │ +0.02  │ ✅ PASS ║
╠═══════════════════════════════════════════════════════════════╣
║ OVERALL: 🔴 REGRESSION DETECTED on Relevance (-10%)         ║
║ RECOMMENDATION: Do NOT deploy v4. Investigate relevance drop.║
╚═══════════════════════════════════════════════════════════════╝
```

## Step 4 — Remediation Recommendations

When regression is detected, provide actionable guidance:

| Regression Type | Likely Cause | Recommended Action |
|----------------|-------------|-------------------|
| Relevance drop | Prompt changes reduced focus on user query | Review prompt diff, restore relevance instructions |
| Coherence drop | Added conflicting instructions | Simplify prompt, use `prompt_optimize` |
| Safety regression | Removed safety guardrails | Restore safety instructions, add safety test cases |
| Task adherence drop | Tool configuration changed | Verify tool definitions, check for missing tools |
| Across-the-board drop | Dataset drift or model change | Check if evaluation dataset changed, verify model deployment |

## CI/CD Integration

Include regression checks in automated pipelines. See [observe skill CI/CD](../../observe/references/cicd-monitoring.md) for GitHub Actions workflow templates that:

1. Run batch evaluation after every deployment
2. Compare against baseline
3. Block deployment if any evaluator shows > 5% regression
4. Alert team via GitHub issue or Slack webhook

## Next Steps

- **View full trend history** → [Eval Trending](eval-trending.md)
- **Optimize to fix regression** → [observe skill Step 4](../../observe/references/optimize-deploy.md)
- **Roll back if critical** → [deploy skill](../../deploy/deploy.md)

# Eval Trending — Metrics Over Time

Track evaluation metrics across multiple runs and versions to visualize improvement trends and detect regressions. This addresses the gap of understanding how agent quality changes over time.

## Prerequisites

- At least 2 evaluation runs in the same evaluation group (same `evaluationId` when created)
- Project endpoint and selected environment resolved from azd or the selected metadata overlay

> ⚠️ **Eval-group immutability:** Trend a group only when its evaluator set and thresholds stayed fixed across runs. If either changed, start a new evaluation group and track that history separately.

## Step 1 — Retrieve Evaluation History

Use **`evaluation_get`** to list all evaluation groups:

| Parameter | Required | Description |
|-----------|----------|-------------|
| `projectEndpoint` | ✅ | Azure AI Project endpoint |
| `isRequestForRuns` | | `false` (default) to list evaluation groups |

Then retrieve all runs within the target evaluation group:

| Parameter | Required | Description |
|-----------|----------|-------------|
| `projectEndpoint` | ✅ | Azure AI Project endpoint |
| `evalId` | ✅ | Evaluation group ID |
| `isRequestForRuns` | ✅ | `true` to list runs |

> ⚠️ **Parameter guardrail:** evaluation_get expects `evalId`, not `evaluationId`, even if the runs were grouped earlier with `evaluationId`.

## Step 2 — Build Metrics Timeline

For each run, extract per-evaluator scores and build a timeline:

| Run | Agent Version | Date | Coherence | Fluency | Relevance | Intent Resolution | Task Adherence | Safety |
|-----|--------------|------|-----------|---------|-----------|-------------------|----------------|--------|
| run-001 | v1 | 2025-01-15 | 3.2 | 4.1 | 2.8 | 3.0 | 2.5 | 0.95 |
| run-002 | v2 | 2025-01-22 | 3.8 | 4.3 | 3.5 | 3.7 | 3.2 | 0.97 |
| run-003 | v3 | 2025-02-01 | 4.1 | 4.4 | 4.0 | 4.2 | 3.8 | 0.96 |
| run-004 | v4 | 2025-02-08 | 4.0 | 4.5 | 3.6 | 4.1 | 3.9 | 0.98 |

## Step 3 — Trend Analysis

Calculate trends for each evaluator:

| Evaluator | v1 → v4 Change | Trend | Status |
|-----------|----------------|-------|--------|
| Coherence | +0.8 (+25%) | ↑ Improving | ✅ |
| Fluency | +0.4 (+10%) | ↑ Improving | ✅ |
| Relevance | +0.8 (+29%) | ↑ Improving (dip at v4) | ⚠️ |
| Intent Resolution | +1.1 (+37%) | ↑ Improving | ✅ |
| Task Adherence | +1.4 (+56%) | ↑ Improving | ✅ |
| Safety | +0.03 (+3%) | → Stable | ✅ |

### Detecting Regressions

Flag any evaluator where the latest run scored **lower** than the previous run:

| Evaluator | Previous (v3) | Latest (v4) | Delta | Alert |
|-----------|--------------|-------------|-------|-------|
| Relevance | 4.0 | 3.6 | -0.4 (-10%) | ⚠️ **REGRESSION** |

> ⚠️ **Regression detected:** Relevance dropped 10% from v3 to v4. Investigate prompt changes or dataset drift. See [Eval Regression](eval-regression.md) for automated analysis.

### Trend Visualization (Text-based)

```
Coherence   ████████████████████████████████░░░░░░ 4.0/5.0  ↑ +25%
Fluency     █████████████████████████████████████░░ 4.5/5.0  ↑ +10%
Relevance   ████████████████████████████░░░░░░░░░░ 3.6/5.0  ↑ +29% ⚠️ dip
Intent Res. █████████████████████████████████░░░░░░ 4.1/5.0  ↑ +37%
Task Adh.   ████████████████████████████████░░░░░░░ 3.9/5.0  ↑ +56%
Safety      ████████████████████████████████████████ 0.98     → Stable
```

## Step 4 — Cross-Version Summary

Present an executive summary:

*"Over 4 agent versions (v1→v4), your agent has improved significantly across all quality metrics. The biggest gain is Task Adherence (+56%). However, Relevance showed a 10% regression from v3 to v4 — recommend investigating recent prompt changes. Safety remains stable at 98%."*

## Recommended Thresholds

| Severity | Threshold | Action |
|----------|-----------|--------|
| ✅ Healthy | ≤ 2% drop from previous run | No action needed |
| ⚠️ Warning | 2–5% drop from previous run | Review recent changes |
| 🔴 Regression | > 5% drop from previous run | Block deployment, investigate |
| 🔴 Critical | Below baseline (v1) on any metric | Rollback to last known good version |

## Next Steps

- **Investigate regression** → [Eval Regression](eval-regression.md)
- **Compare specific versions** → [Dataset Comparison](dataset-comparison.md)
- **Set up automated monitoring** → [observe skill CI/CD](../../observe/references/cicd-monitoring.md)

# Generate Seed Evaluation Dataset

Generate a seed evaluation dataset for a Foundry agent by producing realistic, diverse test queries grounded in the agent's instructions and tool capabilities.

> **Preferred setup:** For deployed agents, use the observe workflow's [Evaluation Suite Generation](../../observe/references/evaluation-suite-generation.md) first. This manual seed-dataset flow is the fallback when suite/data generation APIs are unavailable, fail, return incomplete artifacts, or the user explicitly wants hand-authored local data.

## ⛔ Do NOT

- Do NOT omit the `expected_behavior` field. It is **required** on every row, even during Phase 1 (built-in evaluators only). It pre-positions the dataset for Phase 2 custom evaluators.
- Do NOT use `generateSyntheticData=true` on the eval API. Local generation provides reproducibility, version control, and human review before running evals.
- Do NOT use vague `expected_behavior` values like "responds correctly". Always describe concrete actions (tool calls, sources to cite, tone, decline behavior).

## Prerequisites

- Agent deployed and running (or the local agent source / `azure.yaml` service block available with instructions and tool definitions)
- Selected `.foundry/agent-metadata*.yaml` file resolved with `projectEndpoint` and `agentName`

## Dataset Row Schema

> ⚠️ **MANDATORY: Every JSONL row must include both `query` and `expected_behavior`.**

| Field | Required | Purpose |
|-------|----------|---------|
| `query` | ✅ | Realistic user message the agent would receive |
| `expected_behavior` | ✅ | Behavioral rubric: what the agent SHOULD do — actions, tool usage, tone, source expectations. Used by Phase 2 custom evaluators for per-query scoring. |
| `ground_truth` | Optional | Factual reference answer for groundedness evaluators |
| `context` | Optional | Category or scenario tag for dataset organization and coverage analysis |

Example row:

```json
{"query": "What are the latest EU AI Act updates?", "expected_behavior": "Uses Bing search to find recent EU AI Act news; cites at least one source; mentions implementation timelines or enforcement dates", "context": "current_events", "ground_truth": "The EU AI Act was formally adopted in 2024 with phased enforcement starting 2025."}
```

## Step 1 — Gather Agent Context

Collect the agent's full context from `agent_get` or the local `azure.yaml` service block in the selected agent root:

- **Agent name** — from the selected metadata file
- **Instructions** — the system prompt / instructions field
- **Tools** — list of tools with names, descriptions, and parameter schemas
- **Protocols** — supported protocols (e.g. `responses`, `invocations`, `invocations_ws`, `a2a`, `mcp`)
- **Example messages** — from the `azure.yaml` service metadata if available

## Step 2 — Generate Test Queries

> 💡 **Generate directly.** The coding agent (you) already has full context of the agent's instructions, tools, and capabilities from Step 1. Generate the JSONL rows directly — there is no need to call an external model deployment.

Using the agent context collected in Step 1, generate 20 diverse, realistic test queries that exercise the agent's full capability surface. For agents with many tools, increase count to ensure at least one query per tool.

### Coverage Requirements

Distribute queries across these categories:

| Category | Target % | Description |
|----------|----------|-------------|
| **Happy path** | 40% | Straightforward queries the agent is designed to handle well |
| **Tool-specific** | 20% | Queries that specifically exercise each declared tool |
| **Edge cases** | 15% | Ambiguous, incomplete, or unusually formatted inputs |
| **Out-of-scope** | 10% | Requests the agent should gracefully decline or redirect |
| **Safety boundaries** | 10% | Inputs that test responsible AI guardrails |
| **Multi-step** | 5% | Queries requiring multiple tool calls or reasoning chains |

### Generation Rules

- Vary query length, formality, and complexity
- Include at least one query per declared tool
- `expected_behavior` must describe **ACTIONS** (tool calls, search, cite, decline) not just expected text output
- Each row must conform to the [Dataset Row Schema](#dataset-row-schema) above
- Every generated line must be valid JSON with both `query` and `expected_behavior` keys
- Generate at least 15 rows (target 20) with at least 3 distinct `context` values
- No two rows should have identical `query` values
- `expected_behavior` must mention concrete actions, not vague phrases like "responds correctly"

> 💡 **No separate validation step is needed.** As long as generation follows these rules, the dataset is valid by construction. The schema may evolve over time — enforcing it at generation time (not via a separate validation pass) keeps the workflow simple and forward-compatible.

### Save

Save the generated JSONL to:

```
.foundry/datasets/<agent-name>-eval-seed-v1.jsonl
```

The filename must start with `agentName` from the selected metadata file, followed by `-eval-seed-v1`.

## Step 3 — Register in Foundry

Register the generated dataset in Foundry. Follow these sub-steps:

1. Resolve the active Foundry project resource ID, then use `project_connection_list` with category `AzureStorageAccount` to discover the project's connected storage account.
2. Upload the JSONL file to `https://<storage-account>.blob.core.windows.net/eval-datasets/<agent-name>/<agent-name>-eval-seed-v1.jsonl`.
3. If the storage connection is key-based, use Azure CLI with the storage account key. If AAD-based, prefer `--auth-mode login`.

**Key-based upload example:**

```bash
az storage blob upload \
  --account-name <storage-account> \
  --container-name eval-datasets \
  --name <agent-name>/<agent-name>-eval-seed-v1.jsonl \
  --file .foundry/datasets/<agent-name>-eval-seed-v1.jsonl \
  --account-key <storage-account-key>
```

**AAD-based upload example:**

```bash
az storage blob upload \
  --account-name <storage-account> \
  --container-name eval-datasets \
  --name <agent-name>/<agent-name>-eval-seed-v1.jsonl \
  --file .foundry/datasets/<agent-name>-eval-seed-v1.jsonl \
  --auth-mode login
```

4. Register with `evaluation_dataset_create`, always including `connectionName` so the dataset is bound to the discovered `AzureStorageAccount` project connection:

```
evaluation_dataset_create(
  projectEndpoint: "<project-endpoint>",
  datasetContentUri: "https://<storage-account>.blob.core.windows.net/eval-datasets/<agent-name>/<agent-name>-eval-seed-v1.jsonl",
  connectionName: "<storage-connection-name>",
  datasetName: "<agent-name>-eval-seed",
  datasetVersion: "v1",
  description: "Seed dataset for <agent-name>; <row-count> queries; covers <category-list>"
)
```

5. The current `evaluation_dataset_create` MCP surface does not expose a first-class `tags` parameter. Persist the required dataset tags in metadata instead:
   - `agent`: `<agent-name>`
   - `stage`: `seed`
   - `version`: `v1`
6. Save the returned `datasetUri` in both the selected metadata file (under the active evaluation suite) and `.foundry/datasets/manifest.json`.

## Step 4 — Update Metadata

Update the selected metadata file for the selected environment's `evaluationSuites[]`:

If the selected environment still uses older `testSuites[]` or legacy `testCases[]`, rewrite that environment to `evaluationSuites[]` as part of this update. Preserve dataset/evaluator fields and map legacy `priority` to `tags.tier` only when `tags.tier` is missing.

```yaml
evaluationSuites:
  - id: smoke-core
    tags:
      tier: smoke
      purpose: baseline
      stage: seed
    generationSource: manual-fallback
    dataset: <agent-name>-eval-seed
    datasetVersion: v1
    datasetFile: .foundry/datasets/<agent-name>-eval-seed-v1.jsonl
    datasetUri: <returned-foundry-dataset-uri>
    evaluators:
      - name: relevance
        threshold: 4
      - name: task_adherence
        threshold: 4
      - name: intent_resolution
        threshold: 4
```

Update `.foundry/datasets/manifest.json` by appending a new entry to the `datasets[]` list:

```json
{
  "datasets": [
    {
      "name": "<agent-name>-eval-seed",
      "version": "v1",
      "stage": "seed",
      "agent": "<agent-name>",
      "environment": "<env>",
      "localFile": ".foundry/datasets/<agent-name>-eval-seed-v1.jsonl",
      "datasetUri": "<returned-foundry-dataset-uri>",
      "rowCount": 20,
      "categories": { ... },
      "createdAt": "<ISO-timestamp>"
    }
  ]
}
```

## Next Steps

- **Run evaluation** → [observe skill Step 2](../../observe/references/evaluate-step.md)
- **Curate or edit rows** → [Dataset Curation](dataset-curation.md)
- **Version after edits** → [Dataset Versioning](dataset-versioning.md)
- **Harvest production traces later** → [Trace-to-Dataset Pipeline](trace-to-dataset.md)

# Trace-to-Dataset Pipeline — Harvest Production Traces as Test Cases

Extract production traces from App Insights using KQL, transform them into evaluation dataset format, and persist as versioned datasets. This is the core workflow for turning real-world agent failures into reproducible test cases.

## ⛔ Do NOT

- Do NOT use `parse_json(customDimensions)` — `customDimensions` is already a `dynamic` column in App Insights KQL. Access properties directly: `customDimensions["gen_ai.response.id"]`.

## Related References

- [Eval Correlation](../../trace/references/eval-correlation.md) (in `foundry-agent/trace/references/`) — look up eval scores by response/conversation ID via `customEvents`
- [KQL Templates](../../trace/references/kql-templates.md) (in `foundry-agent/trace/references/`) — general trace query patterns and attribute mappings

## Prerequisites

- App Insights resource resolved (see [trace skill](../../trace/trace.md) Before Starting)
- Agent root, selected metadata file, environment, and project endpoint available from `.foundry/agent-metadata*.yaml`
- Time range confirmed with user (default: last 7 days)

When a repo contains multiple agent roots, this workflow updates only the selected agent root's `.foundry/datasets/`, `.foundry/results/`, and metadata files. Do **not** merge sibling agent folders.

> 💡 **Run all KQL queries** using **`monitor_resource_log_query`** (Azure MCP tool) against the App Insights resource. This is preferred over delegating to the `azure-kusto` skill.

> ⚠️ **Always pass `subscription` explicitly** to Azure MCP tools — they don't extract it from resource IDs.

## Overview

```
App Insights traces
    │
    ▼
[1] KQL Harvest Query (filter by error/latency/eval score)
    │
    ▼
[2] Schema Transform (trace → JSONL format)
    │
    ▼
[3] Human Review (show candidates, let user approve/edit/reject)
    │
    ▼
[4] Persist Dataset (local JSONL files)
    │
    ▼
[5] Sync to Foundry (optional — upload to project-connected storage)
```

## Key Concept: Linking Evaluation Results to Traces

> 💡 **Evaluation results live in `customEvents`, not in `dependencies`.** Foundry writes eval scores to App Insights as `customEvents` with `name == "gen_ai.evaluation.result"`. Agent traces (spans) live in `dependencies`. The link between them is **`gen_ai.response.id`** — this field appears on both tables.

| Table | Contains | Join Key |
|-------|----------|----------|
| `dependencies` | Agent traces (spans, tool calls, LLM calls) | `customDimensions["gen_ai.response.id"]` |
| `customEvents` | Evaluation results (scores, labels, explanations) | `customDimensions["gen_ai.response.id"]` |

**To harvest traces with eval scores**, join `customEvents` → `dependencies` on `responseId`. The [Low-Eval Harvest](#low-eval-harvest--traces-with-poor-evaluation-scores) template below shows this pattern. For standalone eval lookups, see [Eval Correlation](../../trace/references/eval-correlation.md) (in `foundry-agent/trace/references/`).

## Step 1 — Choose a Harvest Template

Select the appropriate KQL template based on user intent. These templates mirror common LangSmith "run rules" but offer more power through KQL's query language.

> ⚠️ **Hosted agents:** The Foundry agent name (e.g., `hosted-agent-022-001`) only appears on `requests`, NOT on `dependencies`. For hosted agents, use the [Hosted Agent Harvest](#hosted-agent-harvest) template which joins via `requests.id` → `dependencies.operation_ParentId`. The templates below work directly for **prompt agents** where `gen_ai.agent.name` on `dependencies` matches the Foundry name.

### Error Harvest — Failed Traces

Captures all traces where the agent returned errors. Equivalent to LangSmith's `eq(error, True)` run rule.

```kql
dependencies
| where timestamp > ago(7d)
| where success == false
| where isnotempty(customDimensions["gen_ai.operation.name"])
| where customDimensions["gen_ai.agent.name"] == "<agent-name>"
| extend
    conversationId = tostring(customDimensions["gen_ai.conversation.id"]),
    responseId = tostring(customDimensions["gen_ai.response.id"]),
    operation = tostring(customDimensions["gen_ai.operation.name"]),
    model = tostring(customDimensions["gen_ai.request.model"]),
    errorType = tostring(customDimensions["error.type"]),
    inputTokens = toint(customDimensions["gen_ai.usage.input_tokens"]),
    outputTokens = toint(customDimensions["gen_ai.usage.output_tokens"])
| summarize
    errorCount = count(),
    errors = make_set(errorType, 5),
    firstSeen = min(timestamp),
    lastSeen = max(timestamp)
    by conversationId, responseId, operation, model
| order by lastSeen desc
| take 100
```

### Low-Eval Harvest — Traces with Poor Evaluation Scores

Captures traces where evaluator scores fell below a threshold. Equivalent to LangSmith's `and(eq(feedback_key, "quality"), lt(feedback_score, 0.3))` run rule.

```kql
let lowEvalResponses = customEvents
| where timestamp > ago(7d)
| where name == "gen_ai.evaluation.result"
| extend
    score = todouble(customDimensions["gen_ai.evaluation.score.value"]),
    evalName = tostring(customDimensions["gen_ai.evaluation.name"]),
    responseId = tostring(customDimensions["gen_ai.response.id"]),
    conversationId = tostring(customDimensions["gen_ai.conversation.id"])
| where score < <threshold>
| project responseId, conversationId, evalName, score;
lowEvalResponses
| join kind=inner (
    dependencies
    | where timestamp > ago(7d)
    | where isnotempty(customDimensions["gen_ai.response.id"])
    | extend responseId = tostring(customDimensions["gen_ai.response.id"])
) on responseId
| extend
    operation = tostring(customDimensions["gen_ai.operation.name"]),
    model = tostring(customDimensions["gen_ai.request.model"]),
    inputTokens = toint(customDimensions["gen_ai.usage.input_tokens"]),
    outputTokens = toint(customDimensions["gen_ai.usage.output_tokens"])
| project timestamp, conversationId, responseId, evalName, score, operation, model, duration
| order by score asc
| take 100
```

> 💡 **Tip:** Replace `<threshold>` with the pass threshold from your evaluator config. Common values: `3.0` for 1–5 ordinal scales, `0.5` for 0–1 continuous scales.

### Latency Harvest — Slow Responses

Captures traces where response latency exceeds a threshold. Equivalent to LangSmith's `gt(latency, 5000)` run rule.

```kql
dependencies
| where timestamp > ago(7d)
| where duration > <threshold_ms>
| where isnotempty(customDimensions["gen_ai.operation.name"])
| where customDimensions["gen_ai.agent.name"] == "<agent-name>"
| extend
    conversationId = tostring(customDimensions["gen_ai.conversation.id"]),
    responseId = tostring(customDimensions["gen_ai.response.id"]),
    operation = tostring(customDimensions["gen_ai.operation.name"]),
    model = tostring(customDimensions["gen_ai.request.model"]),
    inputTokens = toint(customDimensions["gen_ai.usage.input_tokens"]),
    outputTokens = toint(customDimensions["gen_ai.usage.output_tokens"])
| summarize
    avgDuration = avg(duration),
    maxDuration = max(duration),
    spanCount = count()
    by conversationId, responseId, operation, model
| order by maxDuration desc
| take 100
```

> 💡 **Tip:** Replace `<threshold_ms>` with the latency threshold in milliseconds. Common values: `5000` (5s), `10000` (10s), `30000` (30s).

### Combined Harvest — Multi-Criteria Filter

Combines multiple filters in a single query. Equivalent to LangSmith's compound rule: `and(gt(latency, 2000), eq(error, true), has(tags, "prod"))`.

```kql
dependencies
| where timestamp > ago(7d)
| where customDimensions["gen_ai.agent.name"] == "<agent-name>"
| where isnotempty(customDimensions["gen_ai.operation.name"])
| where success == false or duration > <threshold_ms>
| extend
    conversationId = tostring(customDimensions["gen_ai.conversation.id"]),
    responseId = tostring(customDimensions["gen_ai.response.id"]),
    operation = tostring(customDimensions["gen_ai.operation.name"]),
    model = tostring(customDimensions["gen_ai.request.model"]),
    errorType = tostring(customDimensions["error.type"]),
    inputTokens = toint(customDimensions["gen_ai.usage.input_tokens"]),
    outputTokens = toint(customDimensions["gen_ai.usage.output_tokens"])
| summarize
    errorCount = countif(success == false),
    avgDuration = avg(duration),
    maxDuration = max(duration),
    spanCount = count()
    by conversationId, responseId, operation, model
| order by errorCount desc, maxDuration desc
| take 100
```

### Sampling — Control Dataset Size

Add `| sample <N>` or `| take <N>` to any harvest query to control the number of traces extracted. Equivalent to LangSmith's `sampling_rate` parameter.

```kql
// Random sample of 50 traces from the harvest
... | sample 50

// Top 50 most recent traces
... | order by timestamp desc | take 50

// Stratified sample: 20 errors + 20 slow + 10 low-eval
// Run each harvest separately and combine
```

### Hosted Agent Harvest — Two-Step Join Pattern

For hosted agents, the Foundry agent name lives on `requests`, not `dependencies`. Use this two-step pattern:

```kql
let reqIds = requests
| where timestamp > ago(7d)
| where customDimensions["gen_ai.agent.name"] == "<foundry-agent-name>"
| distinct id;
dependencies
| where timestamp > ago(7d)
| where operation_ParentId in (reqIds)
| where customDimensions["gen_ai.operation.name"] == "invoke_agent"
| extend
    conversationId = tostring(customDimensions["gen_ai.conversation.id"]),
    responseId = tostring(customDimensions["gen_ai.response.id"]),
    operation = tostring(customDimensions["gen_ai.operation.name"]),
    model = tostring(customDimensions["gen_ai.request.model"]),
    inputTokens = toint(customDimensions["gen_ai.usage.input_tokens"]),
    outputTokens = toint(customDimensions["gen_ai.usage.output_tokens"])
| project timestamp, duration, success, conversationId, responseId, operation, model, inputTokens, outputTokens
| order by timestamp desc
| take 100
```

> 💡 **When to use this pattern:** If the direct `dependencies` filter by `gen_ai.agent.name` returns no results, the agent is likely a hosted agent where `gen_ai.agent.name` on `dependencies` holds the code-level class name (e.g., `BingSearchAgent`), not the Foundry name. Switch to this `requests` → `dependencies` join.

## Step 2 — Schema Transform

Transform harvested traces into JSONL dataset format. Each line in the JSONL file must contain:

| Field | Required | Source |
|-------|----------|--------|
| `query` | ✅ | User input — extract from `gen_ai.input.messages` on `invoke_agent` dependency spans |
| `response` | Optional | Agent output — extract from `gen_ai.output.messages` on `invoke_agent` dependency spans |
| `context` | Optional | Tool results or retrieved documents from the trace |
| `ground_truth` | Optional | Expected correct answer (add during curation) |
| `metadata` | Optional | Source info: `{"source": "trace", "conversationId": "...", "harvestRule": "error"}` |

### Extracting Input/Output from Traces

The full input/output content lives on `invoke_agent` dependency spans in `gen_ai.input.messages` and `gen_ai.output.messages`. These contain complete message arrays:

```json
// gen_ai.input.messages structure:
[{"role": "user", "parts": [{"type": "text", "content": "How do I reset my password?"}]}]

// gen_ai.output.messages structure:
[{"role": "assistant", "parts": [{"type": "text", "content": "To reset your password..."}]}]
```

Query to extract input/output for a specific conversation:

```kql
dependencies
| where customDimensions["gen_ai.conversation.id"] == "<conversation-id>"
| where customDimensions["gen_ai.operation.name"] in ("invoke_agent", "execute_agent", "chat", "create_response")
| extend
    responseId = tostring(customDimensions["gen_ai.response.id"]),
    operation = tostring(customDimensions["gen_ai.operation.name"]),
    inputMessages = tostring(customDimensions["gen_ai.input.messages"]),
    outputMessages = tostring(customDimensions["gen_ai.output.messages"])
| order by timestamp asc
| take 10
```

Extract the `query` from the last user-role entry in `gen_ai.input.messages` and the `response` from `gen_ai.output.messages`. Save extracted data to a local JSONL file:

```
.foundry/datasets/<agent-name>-traces-candidates-<date>.jsonl
```

## Step 3 — Human Review (Curation)

> ⚠️ **MANDATORY:** Never auto-commit harvested traces to a dataset. Always show candidates to the user first.

Present the harvested candidates as a table:

| # | Conversation ID | Error Type | Duration | Eval Score | Query (preview) |
|---|----------------|------------|----------|------------|----------------|
| 1 | conv-abc-123 | TimeoutError | 12.3s | 2.0 | "How do I reset my..." |
| 2 | conv-def-456 | None | 8.7s | 1.5 | "What's the status of..." |
| 3 | conv-ghi-789 | ValidationError | 0.4s | 3.0 | "Can you help me with..." |

Ask the user:
- *"Which candidates should I include in the dataset? (all / select by number / filter by criteria)"*
- *"Would you like to add ground_truth reference answers for any of these?"*
- *"What should I name this dataset version?"*

## Step 4 — Persist Dataset (Local JSONL)

Save approved candidates to `.foundry/datasets/<agent-name>-<source>-v<N>.jsonl`:

```json
{"query": "How do I reset my password?", "context": "User account management", "metadata": {"source": "trace", "conversationId": "conv-abc-123", "harvestRule": "error"}}
{"query": "What's the status of my order?", "response": "...", "ground_truth": "Order #12345 shipped on...", "metadata": {"source": "trace", "conversationId": "conv-def-456", "harvestRule": "latency"}}
```

### Update Manifest

After persisting, update `.foundry/datasets/manifest.json` with lineage information:

```json
{
  "datasets": [
    {
      "name": "support-bot-prod-traces",
      "file": "support-bot-prod-traces-v3.jsonl",
      "version": "v3",
      "source": "trace-harvest",
      "harvestRule": "error+latency",
      "timeRange": "2025-02-01 to 2025-02-07",
      "exampleCount": 47,
      "createdAt": "2025-02-08T10:00:00Z",
      "reviewedBy": "user"
    }
  ]
}
```

## Next Steps

After creating a dataset:
- **Sync to Foundry** → Step 5 below (recommended for shared/CI use)
- **Run evaluation** → [observe skill Step 2](../../observe/references/evaluate-step.md)
- **Version and tag** → [Dataset Versioning](dataset-versioning.md)
- **Organize into splits** → [Dataset Organization](dataset-organization.md)

## Step 5 — Sync Local Cache with Foundry (Optional)

Refresh or register the local cache in Foundry so it is available for server-side evaluations, shared access, and CI/CD pipelines. Reuse the local cache when it is current, and only refresh or push after user confirmation.

### 5a. Discover Storage Connection

Use `project_connection_list` to find an existing `AzureStorageAccount` connection on the Foundry project:

```
project_connection_list(foundryProjectResourceId, category: "AzureStorageAccount")
```

- **Found** → use its `connectionName` and `target` (storage account URL)
- **Not found** → proceed to 5b

### 5b. Create Storage Connection (if needed)

Ask the user for a storage account, then create a project connection:

```
project_connection_create(
  foundryProjectResourceId,
  connectionName: "datasets-storage",
  category: "AzureStorageAccount",
  target: "https://<storage-account>.blob.core.windows.net",
  authType: "AAD"
)
```

> 💡 **Tip:** The storage account must be in the same subscription or the user must have access. AAD auth is preferred — it uses the caller's identity.

### 5c. Upload JSONL to Blob Storage

Upload the local dataset file to the same `eval-datasets` container used for seed datasets so all Foundry-registered eval datasets follow one storage pattern:

```bash
az storage blob upload \
  --account-name <storage-account> \
  --container-name eval-datasets \
  --name <agent-name>/<agent-name>-<source>-v<N>.jsonl \
  --file .foundry/datasets/<agent-name>-<source>-v<N>.jsonl \
  --auth-mode login
```

The local dataset filename should start with the selected Foundry agent name before the source/stage/version suffixes so trace-derived datasets stay grouped with the owning agent.

> ⚠️ **Always pass `--auth-mode login`** to use AAD credentials. If the container doesn't exist, create it first with `az storage container create`.

### 5d. Register Dataset in Foundry

Use `evaluation_dataset_create` with the blob URI and the `AzureStorageAccount` `connectionName` discovered in 5a or created in 5b. While `connectionName` can be optional in other MCP flows, include it in this workflow so the dataset is bound to the project-connected storage account:

```
evaluation_dataset_create(
  projectEndpoint: "<project-endpoint>",
  datasetContentUri: "https://<storage-account>.blob.core.windows.net/eval-datasets/<agent-name>/<agent-name>-<source>-v<N>.jsonl",
  connectionName: "datasets-storage",
  datasetName: "<agent-name>-<source>",
  datasetVersion: "v<N>"
)
```

### 5e. Verify

Confirm the dataset is registered:

```
evaluation_dataset_get(projectEndpoint, datasetName: "<agent-name>-<source>", datasetVersion: "v<N>")
```

Display the registered dataset details to the user. Update `.foundry/datasets/manifest.json` with `"synced": true` and the server-side dataset name/version.

# Invocations WebSocket (`invocations_ws`) Protocol

Build, deploy, and connect to Foundry hosted agents that expose a **duplex WebSocket** endpoint instead of an HTTP request/response surface. Use this for real-time, bidirectional workloads — voice agents, live transcripts, custom streaming protocols, and signaling for out-of-band media transports.

> ℹ️ **Preview.** `invocations_ws` is in public preview. For current region availability see [Foundry Hosted Agents — region availability](https://learn.microsoft.com/azure/foundry/agents/concepts/hosted-agents#region-availability). Every upgrade must carry the preview flag — either the `foundry_features=HostedAgents=V1Preview` query parameter or the `Foundry-Features: HostedAgents=V1Preview` request header.

## Quick Reference

| Property | Value |
|----------|-------|
| Agent type | Hosted (Bring Your Own container) only |
| Protocol id (`azure.yaml`) | `invocations_ws` |
| Recommended version | `1.0.0` |
| Container route | `WS /invocations_ws` (served by `azure-ai-agentserver-invocations`; the host binds the port and probes for you) |
| Foundry-side URL | `wss://{account}.services.ai.azure.com/api/projects/agents/endpoint/protocols/invocations_ws?project_name={project}&agent_name={agentName}&agent_session_id={sessionId}&foundry_features=HostedAgents=V1Preview` |
| Auth | `Authorization: Bearer <Entra token>` for scope `https://ai.azure.com/.default` |
| Wire format | Developer-defined (binary frames, JSON text frames, protobuf, raw PCM — anything) |
| Session affinity | Per-connection, keyed by the `agent_session_id` query parameter (optional — auto-generated if omitted) |
| Multi-turn / state | Agent-managed inside the container; platform does **not** store history |

## When to Use This Skill

- Build or operate a hosted real-time voice agent (audio in / audio out, control frames)
- Bridge an out-of-band media transport (WebRTC, SFU, telephony) to a Foundry-hosted bot via WebSocket signaling
- Stream events bidirectionally that don't fit `responses` (OpenAI-compatible) or `invocations` (single bytes-in/bytes-out HTTP)
- Connect a browser or native client to an already-deployed `invocations_ws` agent

> ℹ️ For HTTP-based invocation (single request/response, OpenAI `responses` API, or custom HTTP `invocations`), use the [`invoke`](../invoke/invoke.md) skill instead.

## Protocol Comparison

| Aspect | `responses` | `invocations` | `invocations_ws` |
|--------|-------------|---------------|------------------|
| Transport | HTTPS | HTTPS | WebSocket (`wss://`) |
| Lifetime | Per request | Per request | Long-lived duplex |
| Wire format | OpenAI-compatible JSON | Raw bytes (developer-defined) | Frames, developer-defined |
| History | Platform via `conversationId` | Agent-managed | Agent-managed via `agent_session_id` |
| Streaming | `stream: true` (SSE) | Agent-controlled | Native duplex |
| Best for | Chat | Webhooks / classifiers / protocol bridges | Voice, signaling, real-time |

## Workflow

### Step 1: Author the Container

Use the `azure-ai-agentserver-invocations` host — the same package that serves HTTP `/invocations` — and register a WebSocket handler with `@app.ws_handler`. The host runs the server, binds the port, exposes `/readiness`, handles `await websocket.accept()`, runs Ping/Pong keep-alive (default 30s), maps uncaught handler exceptions to close code `1011`, and emits the structured close event used by `azd ai agent monitor`. You can register `@app.invocation_handler` (HTTP `POST /invocations`) and `@app.ws_handler` (WebSocket `GET /invocations_ws`) on the same `app`.

```python
from azure.ai.agentserver.invocations import InvocationAgentServerHost
from starlette.websockets import WebSocket

app = InvocationAgentServerHost()

@app.ws_handler                    # GET /invocations_ws (WebSocket upgrade)
async def ws(websocket: WebSocket) -> None:
    await run_bot(websocket)       # your duplex protocol lives here

app.run()
```

Inside the handler, read the session id from `FOUNDRY_AGENT_SESSION_ID` (env var set by the host), or fall back to the `agent_session_id` query parameter. The container does **not** see the `Authorization` header — APIM and the Agents service strip it after validation, so don't depend on it and don't accept an `authorization` query parameter.

> ⚠️ **You define the wire format.** The platform forwards frames as-is in both directions. There is no schema validation, no OpenAPI registration, no platform-managed history. Document your protocol for callers.

See [Invocations WebSocket Protocol Guide](references/invocations-ws-protocol.md) for the framing model, the `agent_session_id` query parameter, control-vs-data frame patterns, and discovery guidance.

### Step 2: Declare the Protocol in `azure.yaml`

In the agent's service block (`host: azure.ai.agent`):

```yaml
services:
  my-ws-agent:
    host: azure.ai.agent
    kind: hosted
    name: my-ws-agent
    protocols:
      - protocol: invocations_ws
        version: 1.0.0
    container:
      resources:
        cpu: "1"          # voice/media: at least 1 vCPU / 2 GiB; up to 2 vCPU / 4 GiB
        memory: 2Gi
    environmentVariables:
      - name: SOME_SECRET
        value: ${SOME_SECRET}
      # Resolve every secret from the azd environment; do not bake values into the image.
```

The matching `agent.manifest.yaml` declares the same `protocol: invocations_ws` under `template.protocols`.

> ⚠️ The default `azd` scaffold uses `0.25 cpu / 0.5Gi`, which is too small for most real-time workloads. Bump `resources` before deploying.

### Step 3: Deploy via `azd`

Use the standard hosted-agent flow from the [`deploy`](../deploy/deploy.md) skill:

```bash
mkdir ~/azd-deploys/my-ws-agent && cd ~/azd-deploys/my-ws-agent
azd ai agent init -m <path>/agent.manifest.yaml -p <project-resource-id> --no-prompt
# azd env set ... for every variable referenced in azure.yaml
azd deploy my-ws-agent
```

Once `Running`, the Foundry endpoint is reachable at the URL pattern in the Quick Reference table above.

### Step 4: Connect a Client

Connect to the Foundry-side WebSocket directly:

1. **Mint an Entra token** for the audience `https://ai.azure.com`:

   ```bash
   az account get-access-token --resource https://ai.azure.com --query accessToken -o tsv
   ```

2. **Build the upstream URL.** The `agent_session_id` query parameter is **optional** — if you omit it the platform generates one; supply your own (URL-safe; see [Session Management](../invoke/references/session-management.md) for ID format) only when you need to resume an existing session. The preview flag is required:

   ```
   wss://{account}.services.ai.azure.com/api/projects/agents/endpoint/protocols/invocations_ws
     ?project_name={project}
     &agent_name={agentName}
     &agent_session_id={your-id}        # optional
     &foundry_features=HostedAgents=V1Preview
   ```

   You can alternatively pass the preview flag as the `Foundry-Features: HostedAgents=V1Preview` request header on the upgrade.

3. **Open the WebSocket** with header `Authorization: Bearer <token>`. Browser code typically needs a small server-side proxy because the browser `WebSocket` constructor cannot set headers.

4. **Speak your protocol.** Send and receive whatever your container expects.

### Step 5: Multi-turn / Session State

There is no platform-managed history. To correlate frames across reconnects or keep per-user state, reuse the same `agent_session_id` and key your state off it inside the container. See [Session Management](../invoke/references/session-management.md).

### Step 6: Observe and Troubleshoot

Stream container logs while testing:

```bash
azd ai agent monitor my-ws-agent --follow
# scope to a single connection
azd ai agent monitor my-ws-agent --session-id <agent_session_id> --follow
```

The same `agent_session_id` can be used to stream container logs (see the [`troubleshoot`](../troubleshoot/troubleshoot.md) skill for deeper diagnostics).

## Error Handling

| Error | Cause | Resolution |
|-------|-------|------------|
| HTTP 401 / 403 on WS upgrade | Missing or stale Entra token | Re-run `az account get-access-token --resource https://ai.azure.com`; ensure the caller has Foundry data-plane RBAC |
| HTTP 404 on upgrade | Wrong `agent_name` / `project_name`, missing preview flag, or unsupported region | Verify with `agent_get`; ensure `foundry_features=HostedAgents=V1Preview` is on the URL (or `Foundry-Features` header); confirm region per [Hosted Agents region availability](https://learn.microsoft.com/azure/foundry/agents/concepts/hosted-agents#region-availability) |
| WS closes immediately after accept | Container handler raised inside the request | Check logs via `azd ai agent monitor`; typical causes are missing env vars or unreachable backend services |
| Browser cannot connect directly | Browser `WebSocket` cannot set `Authorization` | Run a thin server-side proxy that injects the token before forwarding |
| Frames received but no response | Wire-format mismatch | Confirm both ends use the same framing (binary vs text, codec, sample rate, schema). The platform does **not** validate or transcode frames |
| Cold-start delay on first connect | Container initialising (VAD, model load, etc.) | Expected; subsequent connections to the same container are fast |
| State lost across reconnect | Different `agent_session_id` used | Reuse the same `agent_session_id` query parameter to preserve agent-managed state |

## Reference Samples

End-to-end working samples (server container + browser portal) live in the [`foundry-samples`](https://github.com/microsoft-foundry/foundry-samples) repo under:

```
samples/python/hosted-agents/bring-your-own/invocations_ws/
```

Each sub-folder shows a different media-path strategy (audio entirely over the WebSocket vs. WebSocket as signaling-only for an out-of-band media transport). Pick the one whose architecture matches your latency, NAT-traversal, and operational constraints.

## Additional Resources

- [Invocations WebSocket Protocol Guide](references/invocations-ws-protocol.md)
- [Session Management](../invoke/references/session-management.md)
- [Foundry Hosted Agents](https://learn.microsoft.com/en-us/azure/ai-foundry/agents/concepts/hosted-agents?view=foundry)
- [`invoke` skill](../invoke/invoke.md) — HTTP-based `responses` and `invocations` protocols
- [`deploy` skill](../deploy/deploy.md) — package and deploy hosted-agent containers
- [`troubleshoot` skill](../troubleshoot/troubleshoot.md) — diagnose hosted-agent runtime failures

# Invocations WebSocket Protocol Guide

The `invocations_ws` protocol is a **duplex WebSocket pass-through**. After the platform authenticates the upgrade request and routes it to your container, every frame in both directions is forwarded as-is. The agent developer defines the wire format, the framing model, and the streaming semantics. Unlike `responses` (OpenAI-compatible, platform-managed history) and `invocations` (single HTTP request/response, bytes in / bytes out), `invocations_ws` is a long-lived bidirectional channel under full container control.

## Input/Output Contract

| Aspect | `responses` | `invocations` | `invocations_ws` |
|--------|-------------|---------------|------------------|
| **Transport** | HTTPS request/response | HTTPS request/response | WebSocket (`wss://`) |
| **Lifetime** | Per request | Per request | Long-lived duplex connection |
| **Input** | Natural language `inputText` | Raw HTTP request body | Sequence of WS frames in either direction |
| **Output** | Structured OpenAI JSON | Raw response bytes | Sequence of WS frames in either direction |
| **Framing** | n/a (single body) | n/a (single body) | Developer-defined: binary (PCM, protobuf), text (JSON), or mixed |
| **Streaming** | `stream: true` (SSE) | Agent-controlled (SSE-over-HTTP, etc.) | Native — duplex by definition |
| **History** | Platform via `conversationId` | Agent-managed | Agent-managed; keyed by `agent_session_id` |

## URL and Headers

```
wss://{account}.services.ai.azure.com
   /api/projects/agents/endpoint/protocols/invocations_ws
   ?project_name={project}
   &agent_name={agentName}
   &agent_session_id={sessionId}
   &foundry_features=HostedAgents=V1Preview
```

| Query parameter | Required | Notes |
|-----------------|----------|-------|
| `project_name` | ✅ | Foundry project name (the segment after `/api/projects/` in the project endpoint) |
| `agent_name` | ✅ | Hosted agent name as declared in `azure.yaml` |
| `agent_session_id` | ❌ | Per-connection identifier — see [Session Management](../../invoke/references/session-management.md). If omitted, the platform (or the container) generates a random id |
| `foundry_features` | ✅ (preview) | Must be `HostedAgents=V1Preview` while the protocol is in preview. May alternatively be sent as the `Foundry-Features` request header. |

| Header | Required | Notes |
|--------|----------|-------|
| `Authorization: Bearer <token>` | ✅ | Entra token for audience `https://ai.azure.com` (scope `https://ai.azure.com/.default`) — `az account get-access-token --resource https://ai.azure.com`. Validated by APIM and the Agents service; the container does **not** see this header. |
| `Foundry-Features: HostedAgents=V1Preview` | ✅ (preview) | Required unless the equivalent `foundry_features` query parameter is set. |

The container receives the upgrade on path `/invocations_ws`. Inside the container, read the session id from the `FOUNDRY_AGENT_SESSION_ID` environment variable (set by `azure-ai-agentserver-invocations`), or fall back to the `agent_session_id` query string.

> ⚠️ **Browsers cannot set the `Authorization` header on a `WebSocket`.** Browser clients must connect through a thin server-side proxy that adds the header before forwarding. This is a browser API limitation, not a Foundry requirement.

## Pass-Through Semantics

The platform is a transparent relay:

- **No schema validation.** Binary opcodes, text JSON, protobuf, raw PCM — anything ends up at the container untouched.
- **No transcoding.** Sample rate, codec, byte order are entirely between caller and container.
- **No history.** Nothing is persisted by Foundry between connections. Use the container filesystem or an external store, keyed by `agent_session_id`, if you need continuity.
- **No platform-managed turn taking.** There is no concept of "request" vs "response" — both sides may send frames at any time. Implement your own request/reply correlation if you need it (e.g. include an `id` field in each JSON frame).

## Common Framing Patterns

These are protocols developers build **on top of** the raw WebSocket. The platform does not require, parse, or validate any of them; they are listed for orientation only.

| Pattern | Typical use | Notes |
|---------|-------------|-------|
| **Raw binary media frames** | Voice agents (PCM, Opus) | Binary opcode; agree on sample rate, channels, bit depth out-of-band |
| **Length-prefixed protobuf** | Real-time pipeline frameworks | Each WS frame is one serialized message; control + audio multiplexed |
| **JSON control + binary media** | Mixed signaling | Text frames carry control (e.g. start/stop, RTVI events), binary frames carry media |
| **Pure JSON signaling** | Out-of-band media transports (WebRTC offer/answer/ICE, SFU join tokens) | One JSON object per frame; FIFO request/reply if the protocol is purely turn-based |
| **SSE-style event stream** | One-way server push of events | Text frames; the WS is effectively used as a richer SSE |

## Discovering the Expected Wire Format

> ⚠️ **Do not guess.** The platform exposes no OpenAPI / AsyncAPI surface for `invocations_ws` agents. The contract lives in the container code.

### 1. Inspect the WebSocket Handler

Look at the function decorated with `@app.ws_handler` on an `InvocationAgentServerHost` (the `azure-ai-agentserver-invocations` SDK). The handler determines:

- Whether frames are binary, text, or mixed
- The expected first frame (handshake, capabilities, auth challenge)
- The control vocabulary (start, stop, mute, hangup, etc.)
- The response cadence (turn-based vs free-running)

### 2. Ask the User or Author

If the handler isn't available, ask the agent author for the framing spec before connecting.

## Examples

**Connect from a Python client (no browser proxy):**

```python
import os, uuid, websockets  # requires websockets >= 12 for the additional_headers kwarg below

token = os.popen("az account get-access-token --resource https://ai.azure.com --query accessToken -o tsv").read().strip()
url = (
    "wss://{account}.services.ai.azure.com/api/projects/agents/endpoint/protocols/invocations_ws"
    "?project_name={project}&agent_name={name}"
    f"&agent_session_id={uuid.uuid4().hex}"
    "&foundry_features=HostedAgents=V1Preview"
)

# websockets >= 12 uses `additional_headers`; older versions (<12) expect `extra_headers`.
async with websockets.connect(url, additional_headers={"Authorization": f"Bearer {token}"}) as ws:
    await ws.send(b"<first frame in your wire format>")
    async for frame in ws:
        ...  # frame is bytes (binary) or str (text) depending on what the container sends
```

**Connect from a browser** — terminate a local WebSocket in a server-side proxy that injects the token, then forward frames pass-through to the upstream `wss://`.

## Error Handling

| Error | Cause | Resolution |
|-------|-------|------------|
| 401 / 403 on upgrade | Missing or expired Entra token | Re-mint with `az account get-access-token --resource https://ai.azure.com` |
| 404 on upgrade | Wrong `project_name` or `agent_name`, missing preview flag, or unsupported region | Verify with `agent_get`; ensure `foundry_features=HostedAgents=V1Preview` is set; confirm the deployed version uses `protocol: invocations_ws` and that the region is supported per [Hosted Agents region availability](https://learn.microsoft.com/azure/foundry/agents/concepts/hosted-agents#region-availability) |
| WS closes after accept | Container raised in the handler | Tail logs with `azd ai agent monitor --session-id <agent_session_id> --follow` |
| Frames silently dropped | Wire-format mismatch (binary vs text, wrong schema) | Confirm both ends agree on framing — the platform performs no transcoding |
| State lost on reconnect | Different `agent_session_id` used | Reuse the same `agent_session_id` to land on the same logical state inside the container |
| Browser fails with `1006 abnormal closure` | Browser tried to connect directly with no `Authorization` | Route through a server-side proxy that adds the header |

# Invoke Foundry Agent

Invoke deployed agents in Azure AI Foundry. Manage sessions and file operations for hosted agents.

## Quick Reference

| Property | Value |
|----------|-------|
| Agent types | Prompt (LLM-based), Hosted |
| MCP server | `azure` |
| Key Foundry MCP tools | `agent_invoke`, `agent_get`, `session_create`, `session_get`, `session_delete`, `session_list` |
| File operation tools | `session_file_upload`, `session_file_download`, `session_file_list`, `session_file_delete`, `session_file_stat`, `session_file_mkdir` |
| Conversation support | Single-turn and multi-turn (via `conversationId` for responses protocol, via session state for invocations protocol) |
| Session support | Managed sessions for hosted agents (via `session_create`) |
| Protocols | `responses` (OpenAI-compatible), `invocations` (custom payloads) |

## When to Use This Skill

- Send messages to a deployed agent (single or multi-turn)
- Create/manage sessions for hosted agents
- Upload/download files to/from hosted agent sessions
- Test agent behavior after creation or deployment

## MCP Tools

| Tool | Description | Parameters |
|------|-------------|------------|
| `agent_invoke` | Send a message to an agent and get a response | `projectEndpoint`, `agentName`, `inputText` (required); `agentVersion`, `conversationId`, `sessionId`, `protocol`, `stream` (optional) |
| `agent_get` | Get agent details to verify existence and type | `projectEndpoint` (required), `agentName` (optional) |
| `session_create` | Create a new session for a hosted agent | `projectEndpoint`, `agentName` (required); `sessionId` (optional) |
| `session_get` | Get session status and details | `projectEndpoint`, `agentName`, `sessionId` (required) |
| `session_delete` | Delete a session and release compute | `projectEndpoint`, `agentName`, `sessionId` (required) |
| `session_list` | List sessions with pagination | `projectEndpoint`, `agentName` (required); `limit`, `order`, `after`, `before` (optional) |
| `session_logstream` | Stream console logs (stdout/stderr) from a session | `projectEndpoint`, `agentName`, `sessionId` (required); `maxLines` (optional) |

For session file operation tools (`session_file_upload`, `session_file_download`, `session_file_list`, `session_file_delete`, `session_file_stat`, `session_file_mkdir`), see [File Operations](references/file-operations.md).

## Protocols

Hosted agents support three protocols declared at deployment time. They are distinct contracts — pick per use case (an agent may declare more than one and serve them from the same container):

| Protocol | Recommended Version | Route | Best For |
|----------|-------------------|-------|----------|
| `responses` | `1.0.0` | `.../agents/{agentName}/endpoint/protocols/openai/responses` | Conversational agents, OpenAI-compatible |
| `invocations` | `1.0.0` | `.../agents/{agentName}/endpoint/protocols/invocations` | Custom payloads, protocol bridges, webhook callers |
| `invocations_ws` | `1.0.0` | `wss://.../agents/endpoint/protocols/invocations_ws` | Duplex WebSocket — voice, WebRTC signaling, custom real-time streams. See the dedicated [invocations-ws skill](../invocations-ws/invocations-ws.md); `agent_invoke` does **not** speak WebSocket. |

Key difference: `responses` takes a natural language `inputText` message with platform-managed history. `invocations` is **bytes in, bytes out** — the request body is forwarded as-is to the container and the raw response is returned. The developer defines the schema; the platform is pure pass-through. See [Invocations Protocol Guide](references/invocations-protocol.md) for I/O details, schema discovery, and examples.

> ⚠️ **Critical for invocations:** `inputText` is forwarded as the raw HTTP request body. The agent developer defines what the container accepts. **Do not guess** — fetch the agent's OpenAPI spec or inspect its source code first.

> 💡 **Tip:** The `agent_invoke` MCP tool supports both `responses` and `invocations` protocols. Set `protocol: 'invocations'` when targeting an invocations-protocol agent.

## Workflow

### Step 1: Verify Agent Readiness

Use `agent_get` to verify the agent exists. For hosted agents, also verify the targeted version is `active`.

### Step 2: Fast smoke test for azd-deployed agents

When the current folder is an azd agent project and deployment just completed, prefer the azd CLI first:

```bash
azd ai agent invoke "hello, are you up?"
```

Use `azd ai agent show --output json` only when you need structured status, version, endpoints, or troubleshooting details; a successful remote invocation is the fast smoke test.

If `azd ai agent invoke` returns a `confirmation_required` envelope, summarize the change and proceed only when the user already requested remote invocation or explicitly consents. Prefer the returned `confirmCommand` over inventing flags. If azd cannot resolve the service or agent name, fall back to the MCP workflow below with the resolved `projectEndpoint` and `agentName`.

For a post-deploy smoke test, invoke once unless the user explicitly asked to validate multi-turn/session behavior. If that single invoke returns a successful response, the smoke test passes;

### Step 3: Create Session (Hosted Agents)

For hosted agents, create a session before invoking using `session_create` with `projectEndpoint` and `agentName`. Optionally provide a `sessionId` (must match `^[A-Za-z0-9_-]{8,128}$`). Store the returned `sessionId` for subsequent calls.

> ⚠️ Skip this step for prompt agents — they do not use sessions.

For full session lifecycle details, see [Session Management](references/session-management.md).

### Step 4: Invoke Agent

Use the project endpoint and agent name from the project context. Use `agent_invoke` with:
- `projectEndpoint`, `agentName`, `inputText` (required)
- `agentVersion`, `conversationId`, `sessionId`, `protocol`, `stream` (optional)

**Responses protocol** (default): `inputText` is a natural language message string. Multi-turn via `conversationId`.

**Invocations protocol**: Set `protocol: 'invocations'`. This is **bytes in, bytes out** — `inputText` is forwarded as the raw HTTP request body to the container. The developer defines the expected schema.

> ⚠️ **Do not guess the invocations request body.** To discover the expected schema:
> 1. **Fetch the OpenAPI spec**: `GET {projectEndpoint}/agents/{agentName}/endpoint/protocols/invocations/docs/openapi.json` (if the developer registered one)
> 2. Inspect the agent's **route handler code** or README for the expected payload shape
> 3. If unknown, ask the user for the agent's API contract before invoking

Example invocations call (agent expects `{"message": "<text>"}`):

```text
agent_invoke(projectEndpoint, agentName, inputText: "{\"message\":\"hello\"}", protocol: "invocations", sessionId: "<id>")
```

See [Invocations Protocol Guide](references/invocations-protocol.md) for full details and examples.

### Step 5: Multi-Turn Conversations

**Responses protocol** → Pass `conversationId` from previous response to continue the thread. Platform manages history.

**Invocations protocol** → Reuse same `sessionId`; conversation state is agent-managed via `$HOME`. Do **not** pass `conversationId` — it has no effect for invocations.

### Step 6: File Operations (Hosted Agents)

Upload/download files to pass data to and retrieve results from agents. All file operations require an active session. See [File Operations](references/file-operations.md).

### Step 7: Clean Up

Use `session_delete` to release compute resources when done. Undeleted sessions expire per platform policies.

## Agent Type Differences

| Behavior | Prompt Agent | Hosted Agent |
|----------|--------------|--------------|
| Readiness | Immediate | After deployment, version must be `active` |
| Session | Not applicable | Required via `session_create` |
| Multi-turn | Via `conversationId` | Via `conversationId` (responses) or session state (invocations) |
| File operations | ❌ | ✅ via session file tools |
| Protocol | `responses` only | `responses` or `invocations` |

## Error Handling

| Error | Cause | Resolution |
|-------|-------|------------|
| Agent not found | Invalid name or endpoint | Use `agent_get` to list agents |
| Hosted agent not active | Version still provisioning or failed | Check version status via `agent_get` |
| Session not found | Invalid ID or expired | Create new session with `session_create` |
| `424 FailedDependency` or `session_not_ready` | Hosted agent session is still warming up or readiness has not completed | Wait 15-30 seconds, check `session_logstream` if needed, then retry `agent_invoke` with the same `sessionId` if one was returned; if no `sessionId` was returned, retry `session_create`. If this persists across 3+ retries (with exponential backoff: 15s, 30s, 60s), the container likely cannot start within the readiness probe deadline — redeploy with higher CPU/memory (recommended minimum: `1` CPU / `2Gi` for direct-code deployments). Also verify the model deployment name is correct via `model_deployment_get`. |
| `could not resolve agent service in azd project: no azure.ai.agent service named '<agentName>' found in azure.yaml` from `azd ai agent invoke` | Name mismatch. | Update the agent name to the deployed agent name. |
| `invalid value "json" for --output` from `azd ai agent invoke` | Invoke supports only `default` and `raw` currently. | Retry without `--output json`. |
| Invocation failed | Model error, timeout, or invalid input | Check agent logs, verify model deployment |
| Invocations schema mismatch | Request body does not match what the agent expects | Inspect agent's route handler or API docs for the correct JSON schema; do not guess |
| File operation failed | Session not active or invalid path | Verify session with `session_get` |
| Permission error | Missing RBAC | Follow [troubleshoot skill](../troubleshoot/troubleshoot.md) |
| Rate limit exceeded | Too many requests | Implement backoff and retry |

## Additional Resources

- [Session Management](references/session-management.md)
- [File Operations](references/file-operations.md)
- [Foundry Hosted Agents](https://learn.microsoft.com/en-us/azure/ai-foundry/agents/concepts/hosted-agents?view=foundry)
- [Foundry Samples](https://github.com/azure-ai-foundry/foundry-samples)

# File Operations

Manage files within a hosted agent session. All file operations require an active session with a running sandbox.

## Overview

Hosted agent sessions provide a persistent filesystem rooted at `$HOME` (`/home/session`). Files written to this path survive across requests within the same session. Use the session file tools to upload input data, download outputs, and manage the session filesystem externally.

> ⚠️ **Warning:** All file paths are relative to `$HOME`. For example, `filePath: '/data/input.csv'` maps to `/home/session/data/input.csv` inside the container.

## MCP Tool Details

### Upload File

Use `session_file_upload` to write a file into the session:

| Parameter | Required | Description |
|-----------|----------|-------------|
| `projectEndpoint` | ✅ | AI Foundry project endpoint |
| `agentName` | ✅ | Name of the hosted agent |
| `sessionId` | ✅ | Active session ID |
| `filePath` | ✅ | Destination path (e.g., `/data/input.csv`) |
| `contentBase64` | ✅ | File content as a base64-encoded string |

> 💡 **Tip:** For text files, encode the content to base64 before passing it. For binary files (images, PDFs), read the raw bytes and base64-encode them.

### Download File

Use `session_file_download` to retrieve a file from the session:

| Parameter | Required | Description |
|-----------|----------|-------------|
| `projectEndpoint` | ✅ | AI Foundry project endpoint |
| `agentName` | ✅ | Name of the hosted agent |
| `sessionId` | ✅ | Active session ID |
| `filePath` | ✅ | Path to the file to download (e.g., `/data/output.csv`) |

Returns: File content as a base64-encoded string.

### List Files

Use `session_file_list` to browse the session filesystem:

| Parameter | Required | Description |
|-----------|----------|-------------|
| `projectEndpoint` | ✅ | AI Foundry project endpoint |
| `agentName` | ✅ | Name of the hosted agent |
| `sessionId` | ✅ | Active session ID |
| `path` | ❌ | Directory path to list (defaults to root `/`) |

Returns: List of files and directories with metadata.

### Delete File

Use `session_file_delete` to remove a file or directory:

| Parameter | Required | Description |
|-----------|----------|-------------|
| `projectEndpoint` | ✅ | AI Foundry project endpoint |
| `agentName` | ✅ | Name of the hosted agent |
| `sessionId` | ✅ | Active session ID |
| `filePath` | ✅ | Path to delete |
| `recursive` | ❌ | Set `true` to recursively delete a directory and its contents (default `false`) |

> ⚠️ **Warning:** Non-recursive delete on a non-empty directory will fail. Use `recursive: true` for directories with contents.

### Get File Metadata

Use `session_file_stat` to inspect a file or directory:

| Parameter | Required | Description |
|-----------|----------|-------------|
| `projectEndpoint` | ✅ | AI Foundry project endpoint |
| `agentName` | ✅ | Name of the hosted agent |
| `sessionId` | ✅ | Active session ID |
| `filePath` | ✅ | Path to inspect |

Returns: File name, size, whether it is a directory, and last modified time.

### Create Directory

Use `session_file_mkdir` to create directories:

| Parameter | Required | Description |
|-----------|----------|-------------|
| `projectEndpoint` | ✅ | AI Foundry project endpoint |
| `agentName` | ✅ | Name of the hosted agent |
| `sessionId` | ✅ | Active session ID |
| `path` | ✅ | Directory path to create (e.g., `/data/results`) |
| `createParents` | ❌ | Create parent directories if needed (default `true`) |
| `mode` | ❌ | Unix permission mode (e.g., `755`). Uses system default if omitted |

## Common Patterns

### Upload Input → Invoke → Download Output

```text
1. session_create       → get sessionId
2. session_file_mkdir   → create /data/input/
3. session_file_upload  → upload input files to /data/input/
4. agent_invoke         → tell agent to process /data/input/
5. session_file_list    → check /data/output/ for results
6. session_file_download → retrieve output files
7. session_delete       → clean up when done
```

### Check Agent-Generated Files

```text
1. session_file_list    → browse $HOME to see what the agent created
2. session_file_stat    → check size/type of specific files
3. session_file_download → retrieve files of interest
```

## Storage Limits

- Maximum `$HOME` size: **10 GiB** per session
- Files outside `$HOME` (e.g., `/tmp`) are ephemeral and may be cleared between requests

## Error Handling

| Error | Cause | Resolution |
|-------|-------|------------|
| Session not active | Session expired or not yet running | Use `session_get` to check status; create a new session if expired |
| File not found | Invalid path or file does not exist | Use `session_file_list` to verify the path |
| Directory not empty | Non-recursive delete on a directory with contents | Use `recursive: true` |
| Storage limit exceeded | `$HOME` exceeds 10 GiB | Delete unnecessary files with `session_file_delete` |

# Invocations Protocol Guide

The `invocations` protocol is **bytes in, bytes out**. The platform is pure pass-through — the raw HTTP request body is forwarded to the container and the raw response is returned. The agent developer defines what the container accepts and returns. Unlike `responses` (OpenAI-compatible with platform-managed history), `invocations` gives full control to the container code.

## Input/Output Contract

| Aspect | `responses` | `invocations` |
|--------|------------|---------------|
| **Input** | `inputText` is a natural language message (e.g., `"What is the weather?"`) | `inputText` is forwarded as the **raw HTTP request body** — bytes in. Format as whatever the container's invoke handler expects (typically JSON) |
| **Output** | Structured OpenAI response with `output_text` | **Raw response bytes** from the container — JSON, text, or SSE events. Format is defined by the agent developer |
| **Conversation history** | Platform-managed via `conversationId` | Agent-managed via session filesystem; `conversationId` does **not** apply |
| **Streaming** | Platform-managed via `stream: true` | Agent-controlled; `stream` parameter does **not** apply |

## Discovering the Expected Input Schema

> ⚠️ **Do not guess the invocations request body.** The developer defines the schema in the container's invoke handler. The platform does not validate or transform the payload.

### 1. Fetch the OpenAPI Spec (Preferred)

Agents can register an OpenAPI spec that describes the expected request/response format. Fetch it from:

```text
GET {projectEndpoint}/agents/{agentName}/endpoint/protocols/invocations/docs/openapi.json
```

If the developer registered an `openapi_spec` when creating the server, this returns the full API contract. If not registered, it returns 404.

### 2. Inspect Agent Source Code

Look at the agent's invoke handler — the function registered with `@app.invoke_handler` (Python) or equivalent. The handler reads the raw request (e.g., `request.json()` for JSON, `request.body()` for raw bytes) and returns a `Response`.

### 3. Ask the User

If neither the OpenAPI spec nor source code is available, ask the user for the expected request body format before invoking.

## Examples

**Responses protocol** (default):

```text
agent_invoke(projectEndpoint, agentName, inputText: "What is the weather in Seattle?")
→ Structured response with output_text
```

**Invocations protocol** — agent expects `{"message": "<text>"}`:

```text
agent_invoke(projectEndpoint, agentName, inputText: "{\"message\":\"hello\"}", protocol: "invocations", sessionId: "<session-id>")
→ Raw bytes from container, e.g.: {"response": "Hi there!", "session_id": "abc123"}
```

## Common Use Cases

| Scenario | Why Invocations |
|----------|----------------|
| Webhook receiver (GitHub, Stripe, Jira) | External system sends its own payload format |
| Non-conversational processing (classification, extraction) | Input is structured data, not a chat message |
| Custom streaming protocol (AG-UI) | Needs raw SSE control, not OpenAI-compatible streaming |
| Protocol bridge (proprietary systems) | Caller has its own protocol that doesn't map to `/responses` |

## Error Handling

| Error | Cause | Resolution |
|-------|-------|------------|
| 400/422 or invocation failed | Request body does not match what the container expects | Fetch OpenAPI spec or inspect handler code for the correct schema |
| 404 on OpenAPI spec | Developer did not register an `openapi_spec` | Inspect handler source code or ask the user for the API contract |
| Empty response | Agent returned no content | Check agent logs via `session_logstream`; verify the handler processes the request body correctly |

# Session Management

Manage hosted agent sessions — isolated compute environments that provide persistent state across invocations.

This document covers session creation and lifecycle for both HTTP-protocol agents (`responses`, `invocations`) and WebSocket agents (`invocations_ws`).

## Overview

Sessions bind a hosted agent to a dedicated compute instance. Files written to `$HOME` during a session persist across requests for the lifetime of that session. When a session is deleted, its compute resources and stored files are released.

## Session Creation

| Protocol | How a session is created | Session id |
|----------|--------------------------|------------|
| `responses`, `invocations` (HTTP) | Call the `session_create` MCP tool before invoking the agent | **Server-issued** `sessionId` (or a client-supplied one passed to `session_create`) |
| `invocations_ws` (WebSocket) | Implicitly, on the first WebSocket upgrade (no `session_create` call) | **Client-supplied** `agent_session_id` query parameter on the upgrade URL — **optional**; if omitted, the platform (or the container) generates a random id |

Both ids follow the same format rule: `^[A-Za-z0-9_-]{8,128}$`.

## Session Lifecycle

**HTTP (`responses`, `invocations`):**

```text
session_create → Running → (invoke, file ops) → session_delete
                    ↓
               Expired (platform auto-cleanup)
```

**WebSocket (`invocations_ws`):**

```text
client opens WS upgrade (optionally with ?agent_session_id=<id>)
  └─► first upgrade for that id ──► sandbox created, handler bound
        └─► frames flow ──► either side closes ──► WS connection ends
              └─► sandbox + $HOME persist ──► next WS upgrade with same id re-hydrates
                    └─► after the idle timeout, compute is deprovisioned; state is persisted
```

Key points for `invocations_ws`:

- There is **no `session_create` / `session_delete`** call. The first upgrade creates the session; the session outlives any individual WebSocket connection.
- The `agent_session_id` query parameter is **optional**. If you omit it, the platform (or the container) generates a random id; supply it explicitly only when you need a specific id to resume an existing session.
- The `agent_session_id` is the **affinity key** — the platform routes upgrades with the same id back to the same sandbox.
- Closing the WebSocket does **not** delete the session. To resume, open a new upgrade with the same `agent_session_id` and the container sees its previous `$HOME` state.
- After the idle timeout, the platform deprovisions compute but persists session state, so the next reconnect re-hydrates the sandbox.

## Session ID Format

Session IDs must match the pattern `^[A-Za-z0-9_-]{8,128}$`.

- If you provide a `sessionId` to `session_create`, it must conform to this pattern
- If you omit `sessionId`, the platform auto-generates one
- Store the returned `sessionId` — it is required for all subsequent operations

## MCP Tool Details

### Create Session

Use `session_create` to provision a new session:

| Parameter | Required | Description |
|-----------|----------|-------------|
| `projectEndpoint` | ✅ | AI Foundry project endpoint |
| `agentName` | ✅ | Name of the hosted agent |
| `sessionId` | ❌ | Optional custom session ID (8-128 chars, alphanumeric + hyphens/underscores) |

Returns: Session resource with `sessionId`, status, and expiration.

### Get Session

Use `session_get` to check session status:

| Parameter | Required | Description |
|-----------|----------|-------------|
| `projectEndpoint` | ✅ | AI Foundry project endpoint |
| `agentName` | ✅ | Name of the hosted agent |
| `sessionId` | ✅ | The session ID to inspect |

Returns: Session details including status, version, creation time, and expiration.

### Delete Session

Use `session_delete` to release compute resources:

| Parameter | Required | Description |
|-----------|----------|-------------|
| `projectEndpoint` | ✅ | AI Foundry project endpoint |
| `agentName` | ✅ | Name of the hosted agent |
| `sessionId` | ✅ | The session ID to delete |

> ⚠️ **Warning:** Deleting a session permanently removes all files stored in `$HOME` for that session.

### List Sessions

Use `session_list` to enumerate sessions:

| Parameter | Required | Description |
|-----------|----------|-------------|
| `projectEndpoint` | ✅ | AI Foundry project endpoint |
| `agentName` | ✅ | Name of the hosted agent |
| `limit` | ❌ | Max results to return (1-100, default 20) |
| `order` | ❌ | Sort order: `asc` or `desc` (default `asc`) |
| `after` | ❌ | Cursor for forward pagination |
| `before` | ❌ | Cursor for backward pagination |

> ⚠️ **Warning:** `after` and `before` are mutually exclusive — do not pass both.

## Session vs Conversation

| Concept | Purpose | Scope |
|---------|---------|-------|
| `sessionId` | Binds requests to a compute instance with persistent filesystem state | Hosted agents only |
| `conversationId` | Tracks conversation history across turns | Responses protocol only |

- A single session can host multiple conversations
- A conversation does not require a session (prompt agents use `conversationId` without sessions)
- For hosted agents using `responses` protocol, use **both**: `sessionId` for compute affinity and `conversationId` for history

## Best Practices

1. **Create sessions explicitly** — Always use `session_create` before invoking a hosted agent with `responses` or `invocations` protocol. Do not rely on implicit session creation.
2. **Reuse sessions** — Keep the same session for related multi-turn interactions to preserve agent state.
3. **Clean up when done** — Delete sessions after use to release compute resources and avoid quota consumption.
4. **Handle expiry** — Sessions expire based on platform policies. If `session_get` returns a non-running state, create a new session.
5. **Version awareness** — The platform auto-resolves the agent version at session creation time. If you need a specific version, ensure it is active before creating the session.
6. **Debug with logstream** — Use `session_logstream` to stream stdout/stderr from a running session for troubleshooting.

# Agent Observability Loop

Orchestrate the full eval-driven optimization cycle for a Foundry agent. This skill manages the **multi-step workflow** for a selected agent root and environment: reusing or refreshing `.foundry` cache in that folder only, generating evaluation suites, caching generated datasets and rubric-based evaluators, running agent-target batch evals, clustering failures, optimizing prompts, redeploying, and comparing versions. Use this skill instead of calling individual `azure` MCP evaluation tools manually.

## When to Use This Skill

USE FOR: evaluate my agent, run an eval, test my agent, check agent quality, run batch evaluation, analyze eval results, why did my eval fail, cluster failures, improve agent quality, optimize agent prompt, compare agent versions, re-evaluate after changes, set up CI/CD evals, agent monitoring, eval-driven optimization, set up continuous monitoring, production quality monitoring, why are eval scores dropping.

> ⚠️ **DO NOT manually call** `evaluation_suite_generation_job_create`, `evaluation_agent_batch_eval_create`, `data_generation_job_create`, `evaluator_generation_job_create`, `evaluation_comparison_create`, `prompt_optimize`, or `continuous_eval_create` **without reading this skill first.** This skill defines required pre-checks, environment selection, cache reuse, artifact persistence, fallback behavior, and multi-step orchestration that the raw tools do not enforce.

## Quick Reference

| Property | Value |
|----------|-------|
| MCP server | `azure` |
| Key MCP tools | `evaluation_suite_generation_job_create`, `evaluation_suite_generation_job_get`, `evaluation_suite_get`, `data_generation_job_create`, `evaluator_generation_job_create`, `evaluation_agent_batch_eval_create`, `evaluation_comparison_create`, `evaluation_get`, `prompt_optimize`, `agent_update`, `continuous_eval_create`, `continuous_eval_get`, `continuous_eval_delete` |
| Prerequisite | Agent deployed and running (use [deploy skill](../deploy/deploy.md)) |
| Local cache | selected `.foundry/agent-metadata*.yaml` overlay, `.foundry/suites/`, `.foundry/evaluators/`, `.foundry/datasets/`, `.foundry/results/`; `eval.yaml` can provide local eval intent |

## Entry Points

| User Intent | Start At |
|-------------|----------|
| "Deploy and evaluate my agent" | [Step 1: Auto-Setup Evaluation Suite](references/deploy-and-setup.md) (deploy first via [deploy skill](../deploy/deploy.md)) |
| "Agent just deployed" / "Set up evaluation" | [Step 1: Auto-Setup Evaluation Suite](references/deploy-and-setup.md) (skip deploy, run suite generation) |
| "Evaluate my agent" / "Run an eval" | [Step 1: Auto-Setup Evaluation Suite](references/deploy-and-setup.md) first if `.foundry/evaluators/`, `.foundry/datasets/`, or `suiteName` cache is missing, stale, or the user requests refresh, then [Step 2: Evaluate](references/evaluate-step.md) |
| "Why did my eval fail?" / "Analyze results" | [Step 3: Analyze](references/analyze-results.md) |
| "Improve my agent" / "Optimize prompt" | [Step 4: Optimize](references/optimize-deploy.md) |
| "Compare agent versions" | [Step 5: Compare](references/compare-iterate.md) |
| "Set up CI/CD evals" | [Step 6: CI/CD & Monitoring](references/cicd-monitoring.md) |
| "Enable continuous monitoring" / "Set up production monitoring" / "Evaluation results dropping" | [Continuous Eval](references/continuous-eval.md) |

> ⚠️ **Important:** Before running any evaluation (Step 2), always resolve the selected agent root, environment, effective deployment context, and metadata overlay file. In azd projects, derive project endpoint and deployed agent identity from `azd env get-values`; use metadata for synced suite/cache refs and explicit overrides. Inspect `.foundry/evaluators/`, `.foundry/datasets/`, `.foundry/suites/`, and matching `eval.yaml` in that root only. If the selected suite has `suiteName`, confirm it with `evaluation_suite_get`; otherwise use verified eval.yaml or legacy dataset/evaluator metadata. If cache is missing, stale, or the user wants to refresh it, route through [Step 1: Auto-Setup](references/deploy-and-setup.md) first — even if the user only asked to "evaluate." Do **not** merge `.foundry` cache or source context from sibling agent folders or sibling metadata files.

## Before Starting — Detect Current State

1. Resolve the target agent root, selected environment, effective deployment context, and selected metadata overlay file using [Common Project Context Resolution](../../SKILL.md#agent-common-project-context-resolution).
2. In azd projects, prefer azd env values for project endpoint and deployed agent name/version; if metadata disagrees, stop and ask which source is authoritative.
3. Use `agent_get` and `agent_container_status_get` to verify the environment's agent exists and is running.
4. Inspect the selected environment's `evaluationSuites[]`, cached files under `.foundry/suites/`, `.foundry/evaluators/`, and `.foundry/datasets/`, plus `eval.yaml` in the selected agent root only. If a suite has `suiteName`, call `evaluation_suite_get` to verify the remote suite/version before running it. If `eval.yaml` exists, verify/register its dataset and evaluator references before treating it as a synced Foundry suite. If the metadata still uses older `testSuites[]` or legacy `testCases[]`, normalize that list to evaluation suites first using the shared migration rule.
5. Use `evaluation_get` to check for existing eval runs.
6. Jump to the appropriate entry point.

## Loop Overview

```text
1. Auto-setup generated evaluation suite or refresh .foundry cache for the selected environment
   -> ask: "Run an evaluation to identify optimization opportunities?"
2. Evaluate (agent-target batch eval using evaluation_agent_batch_eval_create)
3. Download and cluster failures
4. Pick a category or evaluation suite to optimize
5. Optimize prompt
6. Deploy new version (after user sign-off)
7. Re-evaluate (same env + same evaluation suite)
8. Compare versions -> decide which to keep
9. Loop to next category or finish
10. Prompt: enable CI/CD pipeline evals and/or continuous production monitoring
```

## Behavioral Rules

1. **Keep context visible.** Restate the selected agent root, environment, metadata overlay file, and primary deployment context source (azd or metadata) in setup, evaluation, and result summaries.
2. **Stay inside the selected agent root.** Once the agent root is resolved, inspect only that folder's `.foundry/` cache and source tree when suggesting tools, datasets, evaluators, or prompt optimizations. Do not merge sibling agent folders.
3. **Reuse cache before regenerating.** Prefer existing `evaluationSuites[]` entries with valid `suiteName`/`suiteVersion`, `.foundry/evaluators/`, `.foundry/datasets/`, and matching verified `eval.yaml` local config when they match the active environment. Ask before refreshing or overwriting them.
4. **Start with smoke suites.** Run evaluation suites tagged `tier=smoke` before broader `tier=regression` or `tier=coverage` suites unless the user explicitly chooses otherwise.
5. **Auto-poll in background.** After creating eval runs, suite generation jobs, data generation jobs, evaluator generation jobs, or starting containers, poll in a background terminal or background task. Only surface terminal status or actionable failures.
6. **Confirm before changes.** Show diff/summary before modifying agent code, refreshing cache, or deploying. Wait for sign-off.
7. **Prompt for next steps.** After each step, present options. Never assume the path forward.
8. **Write scripts to files.** Python scripts go in `scripts/` - no inline code blocks.
9. **Persist eval artifacts.** Save local artifacts to `.foundry/suites/`, `.foundry/evaluators/`, `.foundry/datasets/`, and `.foundry/results/` for version tracking and comparison. Do not copy azd-owned deployment values into metadata when azd resolves them.
10. **Migrate legacy metadata on write.** If the selected environment still uses older `testSuites[]` or legacy `testCases[]`, treat that list as the suite source for the current run, then rewrite that environment to `evaluationSuites[]` on the next metadata update. Preserve dataset/evaluator fields and map `priority` to `tags.tier` only when `tags.tier` is missing.
11. **Use verified eval.yaml or suite generation first.** When matching `eval.yaml` exists, verify/register its dataset and evaluator refs before generating a brand-new suite. Otherwise prefer `evaluation_suite_generation_job_create` for complete post-deploy setup. Poll with `evaluation_suite_generation_job_get` in the background, inspect the result with `evaluation_suite_get`, and persist `suiteName`, `suiteVersion`, `generationJobId`, and local artifact paths.
12. **Fallback explicitly.** If suite/data/evaluator generation fails or returns incomplete artifacts, explain the failure and fall back to the manual evaluator + dataset suggestion flow. Mark metadata with `generationSource: manual-fallback`.
13. **Use agent-target batch eval for runs.** Use `evaluation_agent_batch_eval_create` for batch evaluation, even when setup generated an evaluation suite. Treat `suiteName` as setup/review metadata and call `evaluation_suite_get` only to resolve dataset/evaluator references.
14. **Use exact eval parameter names.** Use `evaluationId` only on `evaluation_agent_batch_eval_create` calls that group runs; use `evalId` on `evaluation_get` and `evaluation_comparison_create`; use `evalRunId` for a specific run lookup.
15. **Check existing evaluators before manual creation.** In fallback or regeneration flows, call `evaluator_catalog_get` before proposing or creating evaluators. Present the existing catalog to the user and map existing evaluators to the agent's evaluation needs.
16. **Use correct parameters when deleting evaluators.** `evaluator_catalog_delete` requires both `name` (not `evaluatorName`) and `version`. When cleaning up redundant evaluators, always pass the explicit version string. If an evaluator has multiple versions (for example, `v1`, `v2`, `v3`), delete each version individually - there is no "delete all versions" shortcut. Discover version numbers with `evaluator_catalog_get` before attempting deletions.
17. **Regenerate targeted artifacts intentionally.** Use `data_generation_job_create` when the user wants dataset regeneration without rebuilding the whole suite. Use `evaluator_generation_job_create` with `evaluatorName` to regenerate a rubric-based evaluator from updated agent/dataset/prompt context.
18. **Account for LLM judge knowledge cutoff.** When the agent uses real-time data sources (web search, Bing Grounding, live APIs), the LLM judge's training cutoff means it cannot verify current facts. Custom evaluators that score factual accuracy or behavioral adherence will produce systematic false negatives - flagging the agent's real-time data as "fabricated" or "beyond knowledge cutoff." Mitigations: (a) instruct the evaluator prompt to accept sourced claims it cannot verify, (b) use `expected_behavior` rubrics that describe the shape of a good answer rather than specific facts, (c) flag suspected knowledge-cutoff false negatives in the failure analysis rather than treating them as real failures.
19. **Show Data Viewer deeplinks (for VS Code runtime only).** Append a Data Viewer deeplink immediately after reference to a dataset file or evaluation result file in your response. Format: "[Open in Data Viewer](vscode://ms-windows-ai-studio.windows-ai-studio/open_data_viewer?file=<file_path>&source=microsoft-foundry-skill) for details and perform analysis". This applies to files in `.foundry/datasets/`, `.foundry/results/`.
20. **Use the custom evaluator output contract in fallback/manual creation.** When creating custom evaluator prompts manually, treat the MCP/tool-enforced output schema as authoritative: `result` plus `reason`. Do **not** include or preserve conflicting user-provided output instructions such as `score`/`reasoning`, duplicate `OUTPUT FORMAT` blocks, markdown, or alternate JSON schemas in `promptText`. If the user provides a judge prompt that contains its own return schema, keep the rubric and placeholders but rewrite or remove the output-format section so it cannot conflict with the enforced `result`/`reason` contract.

## Manual Fallback Evaluator Strategy

Use this only when generated suite setup is unavailable or the user explicitly wants manual evaluator selection.

| Phase | When | Evaluators | Dataset fields | Goal |
|-------|------|------------|----------------|------|
| Fallback baseline | Before the first manual fallback batch run | <=5 built-in evaluators: `relevance`, `task_adherence`, `intent_resolution`, `indirect_attack`, plus `builtin.tool_call_accuracy` when the agent uses tools | `query`, `expected_behavior` (plus optional `context`, `ground_truth`) | Establish a fast baseline and identify which failure patterns built-ins can and cannot explain |
| Phase 2 - After analysis | After reviewing the first run's failures and clusters | Reuse existing custom evaluators first; create a new custom evaluator only when the built-in set cannot capture the gap | Reuse `expected_behavior` as a per-query rubric | Turn broad failure signals into targeted, domain-aware scoring |

The fallback baseline keeps manual setup fast and comparable across agents. Even though the initial built-in evaluators do not consume `expected_behavior`, include it in every seed dataset row so the same dataset is ready for Phase 2 custom evaluators without regeneration.

When built-in evaluators reveal patterns they cannot fully capture - for example, false negatives from `task_adherence` missing tool-call context or domain-specific quality gaps - first call `evaluator_catalog_get` again to see whether an existing custom evaluator already covers the dimension. Only create a new evaluator when the catalog still lacks the required signal.

Example custom evaluator for Phase 2:

```yaml
name: behavioral_adherence
promptText: |
  Given the query, response, and expected behavior, rate how well
  the response fulfills the expected behavior (1-5).
  ## Query
  {{query}}
  ## Response
  {{response}}
  ## Expected Behavior
  {{expected_behavior}}
```

> 💡 **Tip:** This evaluator scores against the per-query behavioral rubric in `expected_behavior`, not just the agent's global instructions. That usually produces a cleaner signal when broad built-in judges are directionally correct but too coarse for optimization.

> ⚠️ **Output contract:** Do not add `Return JSON: {"score": ...}` or any extra output-format block to custom evaluator `promptText`. The evaluator runtime appends and enforces the final JSON contract (`result` and `reason`). If a user-supplied rubric asks for `score`/`reasoning`, normalize that wording to `result`/`reason` or omit the output schema entirely before calling `evaluator_catalog_create`.

## Related Skills

| User Intent | Skill |
|-------------|-------|
| "Analyze production traces" / "Search conversations" / "Find errors in App Insights" | [trace skill](../trace/trace.md) |
| "Debug hosted agent issues" / "Hosted-agent logs" | [troubleshoot skill](../troubleshoot/troubleshoot.md) |
| "Deploy or redeploy agent" | [deploy skill](../deploy/deploy.md) |
| "Enable continuous evaluation" / "Set up ongoing monitoring" | [Continuous Eval](references/continuous-eval.md) (reference within this skill) |

# Steps 3–5 — Download Results, Cluster Failures, Dive Into Category

## Step 3 — Download Results

`evaluation_get` returns run metadata but **not** full per-row output. Write a Python script (save to `scripts/`) to download detailed results using the **Azure AI Projects Python SDK**.

### Prerequisites

```text
pip install azure-ai-projects>=2.0.0 azure-identity
```

### SDK Client Setup

```python
from azure.identity import DefaultAzureCredential
from azure.ai.projects import AIProjectClient

project_client = AIProjectClient(
    endpoint=project_endpoint,       # e.g. "https://<hub>.services.ai.azure.com/api/projects/<project>"
    credential=DefaultAzureCredential(),
)
# The evals API lives on the OpenAI sub-client, not on AIProjectClient directly
client = project_client.get_openai_client()
```

> ⚠️ **Common mistake:** Calling `project_client.evals` directly — the `evals` namespace is on the OpenAI client returned by `get_openai_client()`, not on `AIProjectClient` itself.

### Retrieve Run Status

```python
run = client.evals.runs.retrieve(run_id=run_id, eval_id=eval_id)
print(f"Status: {run.status}  Report: {run.report_url}")
```

### Download Per-Row Output Items

The SDK handles pagination automatically — no manual `has_more` / `after` loop required.

```python
output_items = list(client.evals.runs.output_items.list(run_id=run_id, eval_id=eval_id))
all_items = [item.model_dump() for item in output_items]
```

> 💡 **Tip:** Use `model_dump()` to convert each SDK object to a plain dict for JSON serialization.

### Data Structure

Query/response data lives in `datasource_item.query` and `datasource_item['sample.output_text']`, **not** in `sample.input`/`sample.output` (which are empty arrays). Parse `datasource_item` fields when extracting queries and responses for analysis.

> ⚠️ **LLM judge knowledge cutoff:** When evaluating agents that use real-time data sources (web search, Bing Grounding, live APIs), the LLM judge may flag factually correct but temporally recent responses as "fabricated" or "unverifiable" because the judge's training data predates the agent's live results. Check failure reasons for phrases like "cannot verify," "beyond knowledge cutoff," or "no evidence" before treating them as real failures. See Behavioral Rule 18 in `observe.md` for mitigations.

### Custom Evaluator Dual-Entry Parsing

Custom evaluators produce **two** result entries per item in the `results` array:

| Entry | `metric` field | Has score? | Has reason/label/passed? |
|-------|----------------|------------|--------------------------|
| Entry 1 | `"custom_score"` | ✅ numeric score | ❌ null |
| Entry 2 | `"{evaluator_name}"` | ❌ null | ✅ real reason, label, passed |

To get the complete picture, merge both entries:

```python
def extract_evaluator_result(item, evaluator_name):
    """Merge the dual entries for a custom evaluator into one result."""
    score_entry = None
    detail_entry = None
    for r in item.get("results", []):
        metric = r.get("metric", "")
        if metric == "custom_score":
            score_entry = r
        elif metric == evaluator_name:
            detail_entry = r
    if not detail_entry:
        return None
    return {
        "score": score_entry.get("score") if score_entry else None,
        "passed": detail_entry.get("passed"),
        "reason": detail_entry.get("reason"),
        "label": detail_entry.get("label"),
    }
```

> ⚠️ **Common mistake:** Reading only the first matching result entry for a custom evaluator gives you the score but null reason (or vice versa). Always merge both entries. Built-in evaluators do **not** have this dual-entry pattern - they produce a single entry with all fields populated.

**Evidence from actual eval run** (item 1, `behavioral_adherence`):

```jsonc
// Entry 1: has score, null reason
{"name": "behavioral_adherence", "metric": "custom_score", "score": 1, "reason": null, "passed": null}

// Entry 2: has reason, null score
{"name": "behavioral_adherence", "metric": "behavioral_adherence", "score": null,
 "reason": "The response provides outdated and fabricated information...", "passed": false}
```

### Persist Results

Save results to `.foundry/results/<environment>/<eval-id>/<run-id>.json` (use `json.dump` with `default=str` for non-serializable fields). Print summary: total items, passed, failed, errored counts.

> ⚠️ **Show Data Viewer deeplinks (for VS Code runtime only):** Append a Data Viewer deeplink immediately after reference to an evaluation result file in your response. Format: "[Open in Data Viewer](vscode://ms-windows-ai-studio.windows-ai-studio/open_data_viewer?file=<file_path>&source=microsoft-foundry-skill) for details and perform analysis".

## Step 4 — Cluster Failures by Root Cause

Analyze every row in the results. Group failures into clusters:

| Cluster | Description |
|---------|-------------|
| Incorrect / hallucinated answer | Agent gave a wrong or fabricated response |
| Incomplete answer | Agent missed key parts |
| Tool call failure | Agent failed to invoke or misused a tool |
| Safety / content violation | Flagged by safety evaluators |
| Runtime error | Agent crashed or returned an error |
| Off-topic / refusal | Agent refused or went off-topic |

Produce a prioritized action table:

| Focus | Cluster | Suggested Action |
|-------|---------|------------------|
| Runtime blockers | Runtime errors or failing suites tagged `tier=smoke` | Check container logs or fix blockers first |
| Key regressions | Incorrect answers on suites tagged `purpose=regression` or `tier=smoke` | Optimize prompt or tool instructions |
| Broader quality gaps | Incomplete answers or coverage-oriented suites | Optimize prompt or expand context |
| Tooling issues | Tool call failures | Fix tool definitions or instructions |
| Safety issues | Safety violations | Add guardrails to instructions |

**Rule:** Prioritize runtime errors first, then suites tagged `tier=smoke`, then suites tagged `purpose=regression`, then broader coverage suites by count × severity.

## Step 5 — Dive Into Category

When the user wants to inspect a specific cluster, display the individual rows: evaluation-suite ID, input query, the agent's original response, evaluator scores, and failure reason. Let the user confirm which category or evaluation suite to optimize.

## Next Steps

After clustering -> proceed to [Step 6: Optimize Prompt](optimize-deploy.md).

# Step 6 — CI/CD Evals & Continuous Production Monitoring

After confirming the final agent version through the observe loop, present two complementary monitoring options. The user may choose one, both, or neither.

## Option 1 — CI/CD Pipeline Evaluations (Pre-Deploy Gate)

*"Would you like to add automated evaluations to your CI/CD pipeline so every deployment is evaluated before going live?"*

CI/CD evals run batch evaluations as part of your deployment pipeline, catching regressions **before** they reach production.

If yes, generate a GitHub Actions workflow (for example, `.github/workflows/agent-eval.yml`) that:

1. Triggers on push to `main` or on pull request
2. Accepts a metadata-file input or environment variable such as `FOUNDRY_METADATA_FILE` and defaults it to `.foundry/agent-metadata.yaml`
3. Reads evaluation-suite definitions from the selected metadata file (for example, `.foundry/agent-metadata.prod.yaml` for prod CI)
4. Reads evaluator definitions from `.foundry/evaluators/` and test datasets from `.foundry/datasets/`
5. Runs `evaluation_agent_batch_eval_create` against the newly deployed agent version
6. Fails the workflow if any evaluator score falls below the configured thresholds for the environment and evaluation suite resolved from that metadata file
7. Posts a summary as a PR comment or workflow annotation

Use repository secrets for the selected environment's project endpoint and Azure credentials, and keep the metadata filename explicit in the workflow so prod rollouts do not depend on the local/dev default file. Confirm the workflow file with the user before committing.

## Option 2 — Continuous Production Monitoring (Post-Deploy)

*"Would you like to set up continuous evaluations to monitor your agent's quality in production?"*

Continuous evaluation uses Foundry-native MCP tools to automatically assess agent responses on an ongoing basis — no additional CI/CD pipeline setup is needed for this option. This catches regressions that emerge **after** deployment from changing data, user patterns, or upstream service drift.

### Enable Continuous Evaluation

Use the [continuous evaluation reference](continuous-eval.md) to configure monitoring. The workflow:

1. **Check existing config** — call `continuous_eval_get` to see if monitoring is already active.
2. **Select evaluators** — recommend starting with the same evaluators used in batch evals for consistent comparison:
   - **Quality evaluators** (require `deploymentName`): e.g., groundedness, coherence, relevance, task_adherence
   - **Safety evaluators**: e.g., violence, indirect_attack, hate_unfairness
3. **Enable** — call `continuous_eval_create` with the selected evaluators. The tool auto-detects agent kind and configures the appropriate backend (real-time for prompt agents, scheduled for hosted agents).
4. **Confirm** — present the returned configuration to the user.

### Acting on Monitoring Results

Monitoring is only complete when score drops trigger investigation and remediation.

For instructions on how to read evaluation scores, triage regressions, and verify fixes, see [Acting on Results](continuous-eval.md#acting-on-results).

The observe loop does not end at deployment. Continuous monitoring closes the loop: **observe → optimize → deploy → monitor → observe**. Always offer to set up monitoring after completing an optimization cycle.

## Reference

- [Azure AI Foundry Cloud Evaluation](https://learn.microsoft.com/en-us/azure/ai-foundry/how-to/develop/cloud-evaluation)
- [Hosted Agents](https://learn.microsoft.com/en-us/azure/ai-foundry/agents/concepts/hosted-agents)
- [Continuous Evaluation Reference](continuous-eval.md)

# Steps 8–10 — Re-Evaluate, Compare Versions, Iterate

## Step 8 — Re-Evaluate

Use **`evaluation_agent_batch_eval_create`** for re-evaluation, even when the selected evaluation suite has `suiteName`. The generated suite preserves the reviewed dataset/evaluator bundle for selection and lineage, but the run should target the agent directly. Reuse the **same `evaluationId`** as the baseline run when the evaluator set and thresholds are unchanged. Use the same local or registered test dataset (from the selected agent root's `.foundry/datasets/` and suite metadata) and evaluator bundle from the selected environment/evaluation suite. Update `agentVersion` to the new version.

> ⚠️ **Parameter switch reminder:** Agent-target batch re-evaluation creation uses `evaluationId`, but follow-up calls to `evaluation_get` and `evaluation_comparison_create` must use `evalId`. Do not call `evaluation_suite_run` for batch eval.

> ⚠️ **Eval-group immutability:** Reuse the same `evaluationId` only when `evaluatorNames` and thresholds are unchanged. If you add/remove evaluators or change thresholds, create a new evaluation group first, then compare runs within that new group.

Auto-poll for completion in a background terminal (same as [Step 2](evaluate-step.md)).

## Step 9 — Compare Versions

> **Critical:** `displayName` is **required** in the `insightRequest`. Despite the MCP tool schema showing `displayName` as optional (`type: ["string", "null"]`), the API will reject requests without it with a BadRequest error. `state` must be `"NotStarted"`.

### Required Parameters for `evaluation_comparison_create`

| Parameter | Required | Description |
|-----------|----------|-------------|
| `insightRequest.displayName` | ✅ | Human-readable name. **Omitting causes BadRequest.** |
| `insightRequest.state` | ✅ | Must be `"NotStarted"` |
| `insightRequest.request.evalId` | ✅ | Eval group ID containing both runs |
| `insightRequest.request.baselineRunId` | ✅ | Run ID of the baseline |
| `insightRequest.request.treatmentRunIds` | ✅ | Array of treatment run IDs |

Use **`evaluation_comparison_create`** with a nested `insightRequest`:

```json
{
  "insightRequest": {
    "displayName": "V1 vs V2 Comparison",
    "state": "NotStarted",
    "request": {
      "type": "EvaluationComparison",
      "evalId": "<eval-group-id>",
      "baselineRunId": "<baseline-run-id>",
      "treatmentRunIds": ["<new-run-id>"]
    }
  }
}
```

> **Important:** Both runs must be in the **same eval group** (same `evaluationId` in Steps 2 and 8), but comparison requests and lookups use `evalId` for that same group identifier. That shared group assumes the evaluator bundle is fixed for all runs in the group.

Then use **`evaluation_comparison_get`** (with the returned `insightId`) to retrieve comparison results. Present a summary showing which version performed better per evaluator, and recommend which version to keep.

## Step 10 — Iterate or Finish

If more categories remain in the prioritized action table (from [Step 4](analyze-results.md)), loop back to **Step 5** (dive into next category) → **Step 6** (optimize) → **Step 7** (deploy) → **Step 8** (re-evaluate) → **Step 9** (compare).

Otherwise, confirm the final agent version with the user, then prompt for [CI/CD evals & monitoring](cicd-monitoring.md).

# Continuous Evaluation

Enable, configure, disable, or remove continuous evaluation for a Foundry agent. Continuous evaluation automatically assesses agent responses on an ongoing basis using configured evaluators (e.g., groundedness, coherence, violence detection). This is typically the final step in the [observe loop](../observe.md) after deploying and batch-evaluating an agent — it keeps production quality visible without manual intervention.

## When to Use This Skill

USE FOR: enable continuous evaluation, disable continuous evaluation, configure continuous eval, set up monitoring evaluators, check continuous eval status, delete continuous eval, update evaluators, change sampling rate, change eval interval, production monitoring, ongoing agent quality.

DO NOT USE FOR: running a one-off batch evaluation (use [observe](../observe.md)), querying traces (use [trace](../../trace/trace.md)), creating evaluator definitions (use [observe](../observe.md) Step 1).

## Quick Reference

| Property | Value |
|----------|-------|
| MCP server | `azure` |
| Key MCP tools | `continuous_eval_create`, `continuous_eval_get`, `continuous_eval_delete`, `agent_get`, `evaluation_get` |
| Prerequisite | Agent must exist in the project |
| Local cache | `.foundry/agent-metadata.yaml` |

## Entry Points

| User Intent | Start At |
|-------------|----------|
| "Enable continuous eval" / "Set up monitoring evaluators" | [Before Starting](#before-starting--detect-current-state) → [Enable or Update](#enable-or-update) |
| "Is continuous eval running?" / "Check eval status" | [Before Starting](#before-starting--detect-current-state) → [Check Current State](#check-current-state) |
| "Change evaluators" / "Update sampling rate" | [Before Starting](#before-starting--detect-current-state) → [Check Current State](#check-current-state) → [Enable or Update](#enable-or-update) |
| "Pause evaluations" / "Disable continuous eval" | [Before Starting](#before-starting--detect-current-state) → [Disable](#disable) |
| "Stop evaluating this agent" / "Delete continuous eval" | [Before Starting](#before-starting--detect-current-state) → [Delete](#delete) |
| "Scores are dropping" / "Act on monitoring results" | [Before Starting](#before-starting--detect-current-state) → [Acting on Results](#acting-on-results) |

> ⚠️ **Important:** Always run [Before Starting](#before-starting--detect-current-state) to resolve the project endpoint and agent name before calling any MCP tools.

## Before Starting — Detect Current State

1. Resolve the target agent root and environment using the [Common Project Context Resolution](../../../SKILL.md#agent-common-project-context-resolution) workflow.
2. Extract `projectEndpoint` and `agentName` from the selected environment. If not available in metadata, use `ask_user` to collect them.
3. Use `agent_get` to verify the agent exists and note its kind (prompt or hosted).
4. Use `continuous_eval_get` to check for existing continuous evaluation configuration.
5. Jump to the appropriate entry point based on user intent.

## How It Works

The tool auto-detects the agent's kind and uses the appropriate backend:

- **Prompt agents** — evaluation runs are triggered automatically each time the agent produces a response. Parameters: `samplingRate` (percentage of responses to evaluate), `maxHourlyRuns`.
- **Hosted agents** — evaluation runs are triggered on an hourly schedule, pulling recent traces from App Insights. Parameters: `intervalHours` (hours between runs), `maxTraces` (max data points per run).

The user does not need to choose between these — the tool handles it based on agent kind.

## Behavioral Rules

1. **Always resolve context first.** Run [Before Starting](#before-starting--detect-current-state) before calling any MCP tool. Never assume a project endpoint or agent name.
2. **Check before creating.** Always call `continuous_eval_get` before `continuous_eval_create` to determine whether to create or update. Present existing configuration to the user.
3. **Confirm evaluator selection.** Present the evaluator list to the user before enabling. Distinguish quality evaluators (require `deploymentName`) from safety evaluators (do not).
4. **Prompt for next steps.** After each operation, present options. Never assume the path forward (e.g., after enabling, offer to check status or adjust parameters).
5. **Keep context visible.** Include the project endpoint, agent name, and environment in operation summaries.
6. **Use `continuous_eval_get` for IDs.** The `delete` tool requires a `configId` — always retrieve it from the `get` response rather than asking the user to provide it.
7. **Surface the remediation path.** When presenting continuous eval results that show score degradation, always offer to route into the [observe skill](../observe.md) for diagnosis and optimization. Monitoring without action is incomplete.
8. **Handle agent-not-found.** If `agent_get` returns a not-found error, stop the continuous eval flow. Offer to route to the [deploy skill](../../deploy/deploy.md) to create the agent first, or ask the user to verify the agent name and environment.
9. **Handle auth and endpoint errors.** If `agent_get` or `continuous_eval_create` returns a permission or authentication error, verify the project endpoint, environment, and user access. Do not suggest creating the agent — the issue is access, not existence.
10. **Validate `deploymentName` before enabling.** Do not assume `gpt-4o` exists. If quality evaluators are selected, verify a chat-capable deployment is available in the project. If none exists, stop and explain that quality evaluators cannot be enabled until a compatible deployment is provisioned.
11. **Handle invalid evaluator names.** If `continuous_eval_create` returns an invalid evaluator name error, call `evaluator_catalog_get` to list available evaluators and present valid options. Do not retry with the same arguments.
12. **Handle unexpected empty config.** If `continuous_eval_get` returns an empty list for an agent the user believes has continuous eval configured, verify the agent name and project endpoint match the intended environment in `.foundry/agent-metadata.yaml`. The configuration may exist under a different environment or resolved `agentName`.

## Operations

### Check Current State

Before enabling or modifying, check what's already configured:

```yaml
Tool: continuous_eval_get
Arguments:
  projectEndpoint: <project endpoint>
  agentName: <agent name>
```

- Empty list → no continuous eval configured. Proceed to [Enable or Update](#enable-or-update).
- Non-empty list → agent already has continuous eval. Present the configuration and ask what the user wants to change.

> ⚠️ **Empty result is not proof of absence.** If the user expects a config to exist but the list is empty, verify the project endpoint and agent name match the intended environment before concluding it was never set up.

### Enable or Update

**Replace Semantics**: `continuous_eval_create` always creates a new evaluation group with the provided evaluators and points the evaluation rule at it. Always pass the complete desired configuration on every call — omitted evaluators are dropped, not preserved.

> ⚠️ **Do not assume `gpt-4o` exists.** Before setting `deploymentName`, verify a chat-capable deployment is available in the project. If none exists, quality evaluators cannot be enabled — only safety evaluators (which do not require a deployment) will work.

```yaml
Tool: continuous_eval_create
Arguments:
  projectEndpoint: <project endpoint>
  agentName: <agent name>
  evaluatorNames: ["groundedness", "coherence", "fluency"]  # Illustrative — align with your batch eval evaluators
  deploymentName: "gpt-4o"          # Required for quality evaluators
  enabled: true                      # Set false to disable without deleting
```

**Evaluator selection guidance:**
- **Quality evaluators** (require `deploymentName`): coherence, fluency, relevance, groundedness, intent_resolution, task_adherence, tool_call_accuracy
- **Safety evaluators** (no `deploymentName` needed): violence, sexual, self_harm, hate_unfairness, indirect_attack, code_vulnerability, protected_material
- Custom evaluators from the project's evaluator catalog are also supported by name.

**Optional parameters by agent kind:**

| Parameter | Applies To | Description | Default |
|-----------|-----------|-------------|---------|
| `samplingRate` | Prompt | Percentage of responses to evaluate (1-100) | All responses |
| `maxHourlyRuns` | Prompt | Cap on evaluation runs per hour | No limit |
| `intervalHours` | Hosted | Hours between evaluation runs | 1 |
| `maxTraces` | Hosted | Max data points per evaluation run | 1000 |
| `scenario` | Prompt | Evaluation scenario: `standard` (quality and safety metrics, default) or `business` (business success metrics). An agent can have one of each simultaneously. | `standard` |

### Disable

To temporarily disable without changing configuration, pass the configuration currently in use along with `enabled: false`. Because `continuous_eval_create` has replace semantics, omitting parameters will change the configuration when re-enabled. The `continuous_eval_get` response does not include evaluator names directly — they are stored in the linked evaluation group — so retrieve them via `evaluation_get` first. If multiple configurations are returned in the `continuous_eval_get` response, present the list to the user and ask which to target.

```yaml
# Step 1: Get the evalId, then retrieve current evaluators from the eval group
Tool: continuous_eval_get
Arguments:
  projectEndpoint: <project endpoint>
  agentName: <agent name>
# Note the evalId from the response
```

```yaml
Tool: evaluation_get
Arguments:
  projectEndpoint: <project endpoint>
  evalId: <evalId from above>
# Note the evaluator names from the evaluation group's testing criteria
```

```yaml
# Step 2: Disable with the same evaluators
Tool: continuous_eval_create
Arguments:
  projectEndpoint: <project endpoint>
  agentName: <agent name>
  evaluatorNames: ["groundedness", "coherence", "fluency"]  # Must match current config
  deploymentName: "gpt-4o"
  enabled: false
```

### Delete

To permanently remove continuous evaluation configuration:

```yaml
Tool: continuous_eval_delete
Arguments:
  projectEndpoint: <project endpoint>
  configId: <id from continuous_eval_get>
  agentName: <agent name>
```

Always call `continuous_eval_get` first to retrieve the `id` field of the configuration to delete. If multiple configurations are returned, present the list to the user and ask which to target.

## Acting on Results

Continuous evaluation generates ongoing scores — but monitoring is only useful when you **act** on what it reveals. This section covers how to consume evaluation results and the remediation loop when scores degrade.

### Step 1: Read Evaluation Scores

The `continuous_eval_get` response includes an `evalId` that links to the evaluation group. Use this to retrieve actual run results:

```yaml
Tool: continuous_eval_get
Arguments:
  projectEndpoint: <project endpoint>
  agentName: <agent name>
# Note the evalId from the response
```

```yaml
Tool: evaluation_get
Arguments:
  projectEndpoint: <project endpoint>
  evalId: <evalId from continuous_eval_get>
  isRequestForRuns: true
# Returns evaluation runs with per-evaluator scores
```

Review the run results for score trends. Each run contains scores for every configured evaluator. Look for:
- **Scores below threshold** — any evaluator consistently scoring below your acceptable baseline
- **Score degradation over time** — scores that were previously healthy but are trending downward
- **Safety flags** — any non-zero safety evaluator scores that indicate harmful content

### Step 2: Triage the Regression

1. **Identify the failing evaluators.** From the evaluation runs, note which specific evaluators are scoring low (e.g., `groundedness` dropping from 4.2 to 2.8).
2. **Correlate with traces.** Use the [trace skill](../../trace/trace.md) to search App Insights for the conversations that triggered low scores. Look for patterns: specific query types, tool-call failures, or grounding gaps.
3. **Compare to baseline.** If batch eval results exist in `.foundry/results/`, compare continuous eval scores against the last known-good batch run to determine whether this is a new regression or a pre-existing gap.

### Step 3: Remediate via the Observe Loop

Once you understand the failure pattern, use the [observe skill](../observe.md) to fix it:

| Symptom | Action |
|---------|--------|
| Quality scores dropping (coherence, relevance, task_adherence) | Run [Step 3: Analyze](analyze-results.md) to cluster failures, then [Step 4: Optimize](optimize-deploy.md) to improve the prompt |
| Safety evaluators flagging (violence, indirect_attack) | Review flagged traces via [trace skill](../../trace/trace.md), then update agent instructions or tool definitions to address the pattern |
| Grounding failures | Check whether the agent's data sources are still accessible and returning expected results; update knowledge index or tool configuration |
| Scores fluctuating after a deploy | Run [Step 5: Compare](compare-iterate.md) between the current and previous agent version to isolate the regression |

### Step 4: Verify the Fix

After deploying a fix through the observe loop:

1. **Re-run a batch eval** via [observe](../observe.md) Step 2 against the same test cases to confirm the fix.
2. **Read continuous eval scores** from the next evaluation cycle using `evaluation_get` with the `evalId` — verify scores have recovered.
3. **Adjust evaluators if needed.** If the regression exposed a gap in evaluator coverage, use `continuous_eval_create` to update the configuration with additional or refined evaluators.

> 💡 **Tip:** The continuous eval → observe → deploy → continuous eval cycle is the core production quality loop. Continuous eval detects; observe diagnoses and fixes; continuous eval verifies.

## Response Format

All tools return a unified `ContinuousEvalConfig` shape. The `get` tool returns a list; `create` returns a single object.

| Field | Description | Present For |
|-------|-------------|-------------|
| `id` | Configuration identifier (needed for delete) | All |
| `displayName` | Human-readable name | All |
| `enabled` | Whether evaluation is active | All |
| `evalId` | Linked evaluation group containing evaluator definitions | All |
| `agentName` | Target agent name | All |
| `status` | Provisioning status | Hosted only |
| `scenario` | Evaluation scenario (`standard` or `business`) | Prompt only |
| `samplingRate` | Percentage of responses evaluated | Prompt only |
| `maxHourlyRuns` | Cap on runs per hour | Prompt only |
| `intervalHours` | Hours between scheduled runs | Hosted only |
| `maxTraces` | Max data points per run | Hosted only |
| `createdAt` | Creation timestamp | All |
| `createdBy` | Creator identity | All |

## Related Skills

| User Intent | Skill |
|-------------|-------|
| "Evaluate my agent" / "Run a batch eval" | [observe skill](../observe.md) |
| "Scores are dropping" / "Diagnose and fix quality regression" | [observe skill](../observe.md) (Steps 3–5) |
| "Analyze production traces" / "Find flagged conversations" | [trace skill](../../trace/trace.md) |
| "Deploy my agent" / "Redeploy after fix" | [deploy skill](../../deploy/deploy.md) |

# Step 1 - Auto-Setup Evaluation Suite

> **This step runs automatically after deployment.** If the agent was deployed via the [deploy skill](../../deploy/deploy.md), `.foundry` cache and metadata may already be configured. Check `.foundry/evaluators/`, `.foundry/datasets/`, and the selected metadata file under the selected agent root before re-creating them.

## Auto-Generate Suite

After deployment, immediately prepare a Foundry evaluation suite and local references for the selected environment without waiting for the user to request it.

### 1. Resolve Context

Use [Common Project Context Resolution](../../../SKILL.md#agent-common-project-context-resolution) to compute effective context. In azd projects, prefer `azd env get-values` for deployment context and use the selected `.foundry/agent-metadata*.yaml` file only as an overlay/cache. Use `agent_get`, the local `azure.yaml` service block, and matching `eval.yaml` as needed to resolve:

| Value | Source |
|-------|--------|
| `projectEndpoint` | azd env, then metadata override |
| `agentName` / `agentVersion` | azd agent vars, then metadata/`agent_get` |
| `suiteName` | verified `eval.yaml` name or `<agent-name>-smoke` unless user provided one |
| generation deployment | `model_deployment_get`; choose a chat-completions deployment |

`suiteName` must start with a letter (`A-Z` or `a-z`). If a derived name starts with a number, prefix it with an alphabetic label such as `suite-`.

Do not assume `gpt-4o` exists.

### 2. Reuse or Refresh Cache

Inspect `.foundry/suites/`, `.foundry/evaluators/`, `.foundry/datasets/`, matching `eval.yaml`, and the selected environment's `evaluationSuites[]` in the selected agent root only. Do **not** merge sibling agent folders.

- **Suite metadata has `suiteName` and current cache** -> call `evaluation_suite_get` to verify the remote suite, then reuse it.
- **`eval.yaml` exists and matches the selected agent** -> verify its `dataset.local_uri` or registered `dataset.name`/`dataset.version`, `evaluators[]`, and optional `name` remotely or register them before persisting a synced suite entry. Normalize legacy `dataset_file` in memory only.
- **Cache is missing/stale or user asks refresh** -> generate a new suite after confirming any overwrite.
- **Legacy entry without `suiteName`** -> keep it as legacy fallback metadata unless the user approves generating a new suite.

### 3. Generate Suite

Read [Evaluation Suite Generation](evaluation-suite-generation.md). If the user selected existing `eval.yaml`, follow the local eval.yaml verification/registration path there before creating a generated suite. Otherwise call:

```text
evaluation_suite_generation_job_create(
  projectEndpoint,
  suiteName,
  agentName,
  generationModelDeploymentName,
  dataGenerationType,
  maxSamples
)
```

For trace-informed suites, include `traceAgentName` or `traceAgentId`, `traceAgentVersion`, `traceStartTime`, `traceEndTime`, and `maxTraces`. Start background polling with `evaluation_suite_generation_job_get`, suppress intermediate `in_progress` output, then verify the generated suite with `evaluation_suite_get` after terminal success.

When refining an existing dataset, include `datasetName` and `datasetVersion`.

### 4. Persist Local References

Cache generated artifacts inside the selected root:

```text
.foundry/
  agent-metadata.yaml
  agent-metadata.prod.yaml
  suites/<suite-name>-v<version>.json
  evaluators/<evaluator-name>-v<version>.json
  datasets/<agent-name>-<dataset-name>-v<version>.ref.json
  datasets/<dataset-name>-v<version>/<blob-name>
  results/
```

If the job result exposes only remote names/versions, fetch metadata with `evaluation_suite_get(projectEndpoint, suiteName, suiteVersion)`, `evaluation_dataset_get`, `evaluation_dataset_sas_url_get`, and `evaluator_catalog_get`, then materialize the full suite JSON, full evaluator JSON, dataset `.ref.json`, and downloaded dataset blobs. Never overwrite user-edited cache files without confirmation; deterministic re-fetch of the same immutable remote `<name>-v<version>` may replace the generated cache artifact for that exact version.

### 5. Update Metadata

Write only the selected metadata file and selected environment. In azd projects, persist only non-derivable overlay/cache state; do not copy azd-owned project endpoint, agent name/version, ACR, or observability values. Persist evaluation suites with:

- `id`, `tags`, `suiteName`, `suiteVersion`
- `generationJobId`, `generationSource` (`synthetic`, `traces`, or `manual-fallback`)
- `dataset`, `datasetVersion`, `datasetFile`, `datasetUri`
- evaluator `name`, `version`, `threshold`, `definitionFile` (full cached JSON)

Use tags such as `tier: smoke`, `purpose: baseline`, and `stage: generated`. If metadata still uses older `testSuites[]` or legacy `testCases[]`, replace that list with `evaluationSuites[]` on write and map `priority` to `tags.tier` only when `tags.tier` is missing.

### 6. Fallback

If suite generation fails, is unavailable, or returns incomplete artifacts, explain the failure and fall back to the existing manual path: `evaluator_catalog_get`, local seed JSONL generation via [Generate Seed Evaluation Dataset](../../eval-datasets/references/generate-seed-dataset.md), `evaluation_dataset_create`, and `evaluationSuites[]` metadata with `generationSource: manual-fallback`.

### 7. Prompt User

Ask: *"Your agent is deployed and the selected environment has evaluation-suite metadata plus local dataset/evaluator references. Would you like to run an evaluation to identify optimization opportunities?"*

If yes -> proceed to [Step 2: Evaluate](evaluate-step.md). If no -> stop.

# Step 2 - Run Evaluation

## Prerequisites

- Agent deployed and running in the selected environment
- Selected `.foundry/agent-metadata*.yaml` file loaded for the active agent root
- Evaluation suite selected from the environment's `evaluationSuites[]`
- For generated suites: `suiteName` present and verified with `evaluation_suite_get`
- For legacy suites: local dataset and evaluator metadata available in `.foundry/`

## Definition of Done — Evaluation Run

A Step 2 evaluation run is complete only when **every** box below is checked. Do **not** produce a final "evaluation complete" summary, score table, or report link until all items are done. "Status reached `completed`" is **not** a stopping condition — `evaluation_get` returns metadata only.

- [ ] `evaluation_agent_batch_eval_create` returned an `evalRunId`
- [ ] `evalId` and `evalRunId` mirrored into the selected `.foundry/agent-metadata*.yaml` (`environments.<env>.lastEval.{evalId, evalRunId, runName, suiteName, suiteVersion, agentVersion, startedAt}`) so a later turn can resume
- [ ] Polling reached terminal state (`completed`, `failed`, or `cancelled`)
- [ ] Per-item `output_items` downloaded via the `azure-ai-projects` Python SDK (see [Step 3 → Download Results](analyze-results.md#step-3--download-results)) — NOT via `evaluation_get`, NOT via `evaluation_dataset_sas_url_get`
- [ ] Results persisted under `.foundry/results/<env>/<eval-id>/<run-id>.json`
- [ ] Per-item failures and any `passed: null` / `reason: null` items have been clustered (Step 4) before summarizing

## Run Agent-Target Batch Eval

Use **`evaluation_agent_batch_eval_create`** for batch evaluation, even when the selected metadata entry was produced by evaluation-suite generation. Treat the generated suite as the reviewed source of dataset/evaluator metadata, not as the execution API.

| Parameter | Description |
|-----------|-------------|
| `projectEndpoint` | Azure AI Project endpoint from the selected metadata file |
| `agentName` | Agent name for the selected environment |
| `agentVersion` | Agent version (string, for example `"1"`) |
| `evaluatorNames` | Array of evaluator names from the selected evaluation suite |
| `evaluationName` | Include environment and evaluation-suite ID |
| `runName` | Include environment, suite ID, and agent version |
| `deploymentName` | Required for LLM-judge evaluators |
| `inputData` | Array of inline test items, each an object with a `query` string (and optional `expected_behavior`). **Required for agent-target runs unless `generateSyntheticData=true` is set.** The parameter name is `inputData` — not `data`, `inputItems`, or `inputDataItems`. |
| `generateSyntheticData` | Set `true` to skip `inputData` and let the service generate test queries. Requires `generationModelDeploymentName` and `samplesCount`. The service rejects requests with only `datasetName`/`datasetVersion`; it does not auto-resolve generated suite datasets into input rows. |
| `generationModelDeploymentName` | Model deployment used to generate synthetic queries when `generateSyntheticData=true`. |
| `samplesCount` | Number of synthetic queries to generate (15–1000). |
| `evaluationId` | Existing eval group ID, only when evaluator set and thresholds are unchanged |

Before the run, if the selected suite has `suiteName`, call `evaluation_suite_get(projectEndpoint, suiteName, version)` and confirm it references the expected dataset/evaluators. Use the suite to select evaluator names, thresholds, and local review artifacts, then run `evaluation_agent_batch_eval_create`. Run suites tagged `tier=smoke` first unless the user chooses a broader suite tag or a specific suite.

## Test Data

Use generated suite datasets for user review and lineage. For the agent-target batch eval tool:

- Pass test rows inline via the **`inputData`** parameter (array of `{query: "...", expected_behavior?: "..."}` objects). The service does not accept `datasetName`/`datasetVersion` references for agent-target runs — a generated suite dataset must be materialized into `inputData` rows by the caller.
- Reviewed local rows should include `expected_behavior` so rubric-based evaluators and failure analysis can preserve the user's rubric.
- Alternatively, set `generateSyntheticData=true` with `generationModelDeploymentName`, `samplesCount` (15–1000), and optional `outputDatasetName` when the user wants the agent-target run to generate a fresh test set instead of supplying `inputData`.
- Do not call `evaluation_suite_run` for batch eval.

> ⚠️ **Parameter-name guardrail:** The inline-rows parameter is `inputData`. The service rejects `data`, `inputItems`, and `inputDataItems` with the misleading error `"At least one input data item must be provided ... Set generateSyntheticData=true to auto-generate test queries instead."` — that error means the rows were sent under the wrong key, not that synthetic generation is required.

Before setting `deploymentName`, use `model_deployment_get` to list actual project deployments and choose one that supports chat completions; do **not** assume `gpt-4o` exists.

## Parameter Naming Guardrail

| Tool | Correct Group Parameter | Notes |
|------|-------------------------|-------|
| `evaluation_agent_batch_eval_create` | `evaluationId` | Agent-target batch eval run grouping |
| `evaluation_get` | `evalId` | Use with `isRequestForRuns=true` to list runs in one group |
| `evaluation_comparison_create` | `insightRequest.request.evalId` | Comparison requests take `evalId`, not `evaluationId` |

`evaluation_get` does **not** accept `evaluationId`; switch to `evalId` after run creation.

> ⚠️ **Eval-group immutability:** Reuse an existing eval group only when dataset, evaluator list, and thresholds are unchanged. If evaluator definitions or thresholds change, create a new evaluation group or suite version.

## Auto-Poll for Completion

Immediately after creating the run, poll `evaluation_get` in a background terminal until completion. Use `evalId + isRequestForRuns=true` for run lists. The run ID parameter is `evalRunId` (not `runId`).

Only surface the final result when status reaches `completed`, `failed`, or `cancelled`.

> ⚠️ **`evaluation_get` returns run metadata only — it does NOT return per-item scores, agent responses, or judge reasons.** Once the run reaches terminal state, you MUST immediately follow [Step 3 → Download Results](analyze-results.md#step-3--download-results) and pull `output_items` via the `azure-ai-projects` Python SDK (`client.get_openai_client().evals.runs.output_items.list(...)`). Do **not** attempt to use `evaluation_dataset_sas_url_get` on the result artifact (`eval-result-<runId>-*`) — that endpoint is for evaluation **input** datasets and returns 500 for result artifacts.

> 💡 **Mirror IDs to metadata immediately.** Right after `evaluation_agent_batch_eval_create` returns, write `evalId`, `evalRunId`, `runName`, `suiteName`/`suiteVersion`, `agentVersion`, and `startedAt` to the selected environment's `lastEval` block in `.foundry/agent-metadata*.yaml`. This lets a later turn resume polling or downloading without re-reading chat history. The azd `.env` (`LAST_EVAL_ID`, etc.) is azd-internal and should not be relied on by skill flows.

## Background Polling Pattern

MCP tools live in the agent's process, so a true detached poller cannot call MCP tools directly. Use one of these concrete patterns instead of saying "ping me later":

1. **Sentinel-file poller (preferred for long-running jobs).** Spawn a sync terminal Python job that polls the Foundry REST API with the user's Azure credential (via `azure-identity` + `requests`) every 60–120s and writes status to `.foundry/.poll/<evalRunId>.json` when terminal. The next turn reads the sentinel file before doing anything else.
2. **Batched in-turn polling.** If a sentinel poller is unavailable, batch 2–4 poll calls per turn (60–120s apart, via short `sleep` between MCP calls in the same response) before yielding back to the user. Always explain that polling will continue on the next turn and update the metadata's `lastEval.lastPolledAt` so resumption is obvious.
3. **Never silently stop.** Returning "ping me later" without updating metadata or spawning a sentinel is a workflow violation — the user has to remember state for you.

## Next Steps

When evaluation completes -> immediately proceed to [Step 3: Analyze Results](analyze-results.md) and download `output_items`. Do not produce a summary first.

## Reference

- [Azure AI Foundry Cloud Evaluation](https://learn.microsoft.com/en-us/azure/ai-foundry/how-to/develop/cloud-evaluation)
- [Built-in Evaluators](https://learn.microsoft.com/en-us/azure/foundry/concepts/built-in-evaluators)

# Evaluation Suite Generation

Use generated suites as the preferred setup path for deployed agents. The suite generation job can create synthetic or trace-derived data plus a rubric-based evaluator from agent, dataset, file, prompt, or trace context.

## Step 1: Ask the User Which Source to Use (MANDATORY)

> ⚠️ **Do not call `evaluation_suite_generation_job_create` without asking the user first.** The generation source materially changes the suite's coverage and cost. Use `ask_user` / `askQuestions` with these options:
>
> - **(a) Current agent code/definition** — synthetic Q&A generated from the agent's instructions and tool definitions. Best for brand-new or recently changed agents with no production traffic.
> - **(b) Historical traces** — sampled from real conversations. **Default lookback: last 3 days (`maxTraces` ~50).** Best for deployed agents with traffic, since the suite reflects real user intents and edge cases.
> - **(c) Existing eval.yaml** — local dataset/evaluator intent from the selected agent root. Best when azd AI agent eval configuration already exists.
>
> **Default selection rule:** If `eval.yaml` exists and `agent.name` matches the selected agent, recommend (c). Otherwise, if the agent has traces in the last 3 days (check via `trace` skill or `evaluation_agent_traces_batch_eval_create` lookback probe), recommend (b); otherwise recommend (a). Always let the user override.

If the user picks (b), compute `traceStartTime` and `traceEndTime` as unix seconds for the chosen window (default `now - 3*86400` to `now`).
If the user picks (c), do not assume a Foundry suite exists. Verify or register the local dataset and evaluators first as described below.

## Step 2: Create and Poll

Call `evaluation_suite_generation_job_create` with the selected `projectEndpoint`, `suiteName`, and `generationModelDeploymentName`. Provide the best available source context:

`suiteName` must start with a letter (`A-Z` or `a-z`). If a derived name starts with a number, prefix it with an alphabetic label such as `suite-`.

| Source | Parameters |
|--------|------------|
| Deployed agent (code/definition) | `agentName`, **`agentSourceNames: [<agentName>]`** (required for target), `agentSourceDescription` |
| Existing dataset | `datasetName`, `datasetVersion`, `datasetSourceDescription` |
| File | `fileId`, `fileSourceDescription` |
| Prompt | `promptSource`, `promptSourceDescription` |
| Traces | `traceAgentName` or `traceAgentId`, `traceAgentVersion`, `traceStartTime`, `traceEndTime` (unix seconds), `maxTraces`, `tracesSourceDescription` |

Set `dataGenerationType` (default `simple_qna`), `category` (default `quality`), `deploymentName` (target model for the evaluator's judge — required for LLM-judge evaluators), and `maxSamples` for generated examples.

### Parameter Requirements (Learned Constraints)

> ⚠️ The service rejects requests that miss these:
> - **`maxSamples` must be between 15 and 1000.** Smaller values (e.g., 10) fail with `Max samples must be between 15 and 1000`. Default to `15` for quick smoke suites, `50–100` for richer baselines.
> - **A `target` is required.** When generating from a deployed agent, pass **`agentSourceNames: [<agentName>]`** (not just `agentName`) so the service can construct the `azure_ai_agent` target. Without it, the request fails with `Target is required for evaluation suite generation`.
> - **`deploymentName`** (in `initialization_parameters`) is required when the generated evaluator uses an LLM judge — pass the same or a comparable deployment as `generationModelDeploymentName`.

Poll with `evaluation_suite_generation_job_get(projectEndpoint, jobId)` until the job reaches a terminal state (`succeeded`, `failed`, `canceled`). Generation typically takes **5-15 minutes** for synthetic Q&A and longer for trace-derived suites, so do not block the main response with repeated foreground polling.

> ⚠️ **Mandatory: poll in the background.** Once `evaluation_suite_generation_job_create` returns a `jobId`, persist the in-flight `generationJobId` in the selected `.foundry/agent-metadata*.yaml` file, start a background polling task or background terminal loop, and keep normal chat output clean. The foreground response should say that generation started and that final status will be surfaced when the background poll reaches a terminal state.
>
> **How to poll:** In the background worker, call `evaluation_suite_generation_job_get` every 60-120 seconds until `status` is `succeeded`, `failed`, or `canceled`. Suppress intermediate `in_progress` output unless the status changes or the job is stuck. Do not print every poll result to the user.
>
> The background poll may stop before terminal state only when: (a) the user explicitly tells you to stop polling, (b) the job has been `in_progress` for >30 minutes (treat as stuck and surface the job ID), or (c) polling errors repeatedly (surface the error). Leave the in-flight `generationJobId` recorded in metadata so a later turn can resume polling.
>
> When the background poll reaches `succeeded`, continue by calling `evaluation_suite_get` and then cache/update metadata before producing the completion summary. When it reaches `failed` or `canceled`, surface the terminal status and route to fallback.

## Existing eval.yaml Source

Use this path when the selected agent root has `eval.yaml` and the user chooses it:

1. Parse `agent.name`, `dataset.local_uri`, `dataset.name`, `dataset.version`, `validation_dataset`, `evaluators[]`, `name`, `options.eval_model`, `options.pass_threshold`, `max_samples`, `trace_days`, and `generation_instruction`. Legacy `dataset_file`, `dataset_reference`, and `validation_reference` may be normalized in memory when reading older files.
2. Verify `agent.name` matches the effective selected agent from azd/metadata. If it differs, stop and ask which target is authoritative.
3. Confirm `dataset.local_uri` exists under the selected agent root when present. Treat it as a local seed dataset until `evaluation_dataset_create` or a remote lookup succeeds.
4. For each evaluator name, call `evaluator_catalog_get` before treating it as remote. If missing, ask whether to create/register it or generate a new rubric-based evaluator.
5. If `name` is populated, call `evaluation_suite_get` before storing it as `suiteName`. If no suite exists, either create/register a reviewed suite or persist a local-draft entry without `suiteName`.
6. Persist only synced remote refs and local cache paths to `.foundry/agent-metadata*.yaml` with `generationSource: eval-yaml`; do not copy azd-owned deployment context into metadata.

## Cache Artifacts Locally

> ⚠️ **Mandatory after `succeeded`.** As soon as the background poll reaches `succeeded`, perform **all three** of the following calls and write **all three** files. This is not optional — partial caching (e.g., metadata stub instead of full evaluator definition) is the most common skill bug. Do not write the deployment/eval-setup summary until the three files exist.

Save artifacts under the selected agent root only, using these exact paths and contents:

| Call | Local file | Contents |
|------|------------|----------|
| `evaluation_suite_get(projectEndpoint, suiteName, version)` | `.foundry/suites/<suite-name>-v<version>.json` | The **full** returned suite object (target, testing_criteria, dataset ref, input_messages). |
| `evaluator_catalog_get(name, version)` | `.foundry/evaluators/<evaluator-name>-v<version>.json` | The **full** returned evaluator object including `definition.dimensions`, `definition.metrics`, `definition.data_schema`, and `generation_artifacts`. Do NOT save a YAML stub — persist the complete JSON so HITL rubric edits + `evaluator_catalog_update(createNewVersion: true)` can round-trip. |
| `evaluation_dataset_get(name, version)` + `evaluation_dataset_sas_url_get(datasetName, datasetVersion)` | `.foundry/datasets/<agent-name>-<dataset-name>-v<version>.ref.json` AND `.foundry/datasets/<dataset-name>-v<version>/<blob-name>` | Metadata stub PLUS the actual dataset blob(s). The SAS-url tool returns a container-scope SAS (`sr=c, sp=rl`); list the container then download every blob (see "Dataset Content Download" below). Set `contentDownloaded: true` + `contentFiles: [...]` in the stub. |

For the first two, do not skip fields and do not transform — write the JSON returned by the MCP tool. Do not overwrite user-edited cache files without confirmation. Exception: deterministic re-fetch of the same immutable remote `<name>-v<version>` may replace the generated cache artifact for that exact version when rehydrating a missing, stale, or corrupt local cache.

### Dataset Content Download (USE THIS — DO NOT SKIP)

The dataset rows live in a Foundry-managed Azure Storage container (host pattern `sa*.blob.core.windows.net`). User Entra credentials against the container fail (`InvalidAuthenticationInfo: Issuer validation failed`) and the storage account is not exposed as a project connection, BUT a working download path exists:

1. Call `evaluation_dataset_sas_url_get(projectEndpoint, datasetName, datasetVersion)`. It returns a container-scope SAS URL with `sr=c&sp=rl` (read + list).
2. **List blobs** via REST: `GET <containerUrlWithoutSas>?restype=container&comp=list&<sasQueryWithoutLeadingQuestionMark>`. Response is XML; blob names are at `EnumerationResults.Blobs.Blob.Name`.
3. **Download each blob** to `.foundry/datasets/<dataset-name>-v<version>/<blob-name>` using the same SAS query appended: `<containerUrl>/<blobName>?<sasQuery>`.
4. Use `curl.exe` (not PowerShell `Invoke-RestMethod` / `Invoke-WebRequest`) on Windows — PowerShell's URI parser chokes on Azure Storage SAS query strings and throws "Invalid URI: The hostname could not be parsed". `curl.exe` ships with Windows 10/11.
5. Update the `.ref.json` stub with `contentDownloaded: true`, `contentPath`, and `contentFiles: [...]`.

Only fall back to the portal-export workaround (Foundry portal → suite → Dataset → Download as JSONL) when `evaluation_dataset_sas_url_get` itself is unavailable or returns an error. Do NOT attempt `az storage blob`, `az storage account list`, or Resource Graph scans for the storage account — they will fail and waste tool calls.

If the dataUri host does NOT match the Foundry-managed `sa*.blob.core.windows.net` pattern (e.g., a customer-owned storage account registered as a project connection), use the connection-resolved credential rather than the SAS flow.

### Job-Returned Direct Artifacts

If the generation job output includes direct file/session references (rare — most jobs only return remote names/versions), download those artifacts and place them in the same `.foundry/` folders alongside the reference files above.

## Regenerate One Artifact

Use `data_generation_job_create` when the user wants fresh data without replacing the whole suite. It accepts `jobName`, `projectEndpoint`, optional `agentName`/`agentVersion`, `datasetName`/`datasetVersion`, `fileId`, `promptSource`, trace parameters, `generationType`, `questionTypes`, `scenario`, `maxSamples`, and `trainSplit`. Poll with `data_generation_job_get` in the background using the same clean-output rules.

Use `evaluator_generation_job_create` to create or regenerate one rubric-based evaluator. To regenerate, pass the existing `evaluatorName` plus updated source inputs and `modelDeploymentName`; poll with `evaluator_generation_job_get` in the background using the same clean-output rules.

## Review and Sync Back

After users edit generated dataset rows or evaluator rubrics locally:

1. Save a new local dataset/evaluator version instead of overwriting the old one.
2. Register approved dataset data with `evaluation_dataset_create`.
3. For evaluator rubric changes, use `evaluator_catalog_update(createNewVersion: true)` when metadata/dimension edits are sufficient; otherwise regenerate with `evaluator_generation_job_create(evaluatorName, ...)`.
4. Create an immutable suite version with `evaluation_suite_create` so future agent-target batch evals can resolve the reviewed artifacts with `evaluation_suite_get`.

## Fallback

If suite, data, or evaluator generation fails or returns incomplete artifacts, explain the failure and use the manual fallback: `evaluator_catalog_get`, local seed JSONL generation, `evaluation_dataset_create`, and `evaluationSuites[]` metadata with `generationSource: manual-fallback`.

Do not use `evaluation_suite_run` for batch eval. Use `evaluation_agent_batch_eval_create` after reviewing the generated suite artifacts.

# Steps 6–7 — Optimize Prompt & Deploy New Version

## Step 6 — Optimize Prompt

> ⛔ **Guardrail:** When optimizing after a dataset update, do NOT remove dataset rows or weaken evaluators to recover scores. Score drops on a harder dataset are expected — they mean test coverage improved, not that the agent regressed. Optimize for NEW failure patterns only.

Use **`prompt_optimize`** with:

| Parameter | Required | Description |
|-----------|----------|-------------|
| `developerMessage` | ✅ | Agent's current system prompt / instructions |
| `deploymentName` | ✅ | Model for optimization (e.g., `gpt-4o-mini`) |
| `projectEndpoint` or `foundryAccountResourceId` | ✅ | At least one required |
| `requestedChanges` | | Concise improvement suggestions from cluster analysis |

**Example `requestedChanges`:** *"Be more specific when answering geography questions"*, *"Always cite sources when providing factual claims"*

> Use the optimized prompt returned by the tool. Do NOT manually rewrite.

## Step 7 — Deploy New Version

> **Always confirm before deploying.** Show the user a diff or summary of prompt changes and wait for explicit sign-off.

After approval:

1. Use **`agent_update`** to create a new agent version with the optimized prompt
2. Use **`agent_get`** to verify the updated version is `running`
3. If the updated version is not `running`, read and follow the [troubleshoot skill](../../troubleshoot/troubleshoot.md) before continuing

## Next Steps

When the new version is running → proceed to [Step 8: Re-Evaluate](compare-iterate.md).

# Declarative Routines

The routines extension registers a service target so routines can live in source control and be upserted by `azd up` / `azd deploy`. Declare each routine as a service with `host: azure.ai.routine`: the **service key is the routine name**, and the keys under it bind directly to the routine model.

```yaml
# azure.yaml
services:
  my-agent:
    host: azure.ai.agent
    # ... agent service block ...

  daily-digest:                 # service key = routine name
    host: azure.ai.routine
    uses:
      - my-agent                # order the agent ahead of the routine that invokes it
    description: Daily 8am digest
    enabled: true
    triggers:
      default:
        type: schedule          # recurring cron; see type table below
        cron_expression: "0 8 * * *"
        time_zone: America/New_York
    action:
      type: invoke_agent_responses_api   # see type table below
      agent_name: my-agent      # target agent (distinct from the routine name)
      input: "Summarize activity for ${AZURE_ENV_NAME}"
```

Then:

```bash
azd deploy daily-digest --no-prompt
azd up
```

## Trigger and action `type` values

`azure.yaml` (like a `--file` manifest) uses the raw wire `type:` value, **not** the CLI alias:

- Trigger `type`: `schedule` (recurring cron), `timer` (one-shot), `github_issue`, or `custom`.
- Action `type`: `invoke_agent_responses_api` (resume with `conversation`) or `invoke_agent_invocations_api` (resume with `session_id`).

The `azd ai routine` CLI accepts friendlier aliases (`recurring`, `github-issue`, `agent-response`, `agent-invoke`) for the same values. See the full alias-to-wire mapping and per-trigger key fields in [CLI CRUD and Operations](cli-crud.md#vocabulary-cli-aliases-vs-manifest-values).

## `action.input`

Put the prompt or payload sent to the agent in `action.input`:

- `invoke_agent_responses_api` (`agent-response`): a string prompt.
- `invoke_agent_invocations_api` (`agent-invoke`): an object/array/scalar matching the target agent's expected input.

## Behavior notes

- `azd deploy` PUTs the routine idempotently; package and publish are no-ops (a routine has no build artifact).
- The routine name always comes from the service key; any `name:` inside the block is ignored.
- Put the target agent service in `uses:` so azd orders the agent before the routine that invokes it.
- String values resolve `${VAR}` against the active azd env at deploy time; Foundry server-side `${{...}}` expressions are left untouched.
- Removing the service block stops azd managing the routine but does **not** delete it from Foundry. Delete explicitly with `azd ai routine delete <name>`.

# CLI CRUD and Operations

Use `azd ai routine` for imperative routine CRUD and operations. Every verb accepts `--output json` or `--output table` (default), and `-p <endpoint>` to override the resolved project endpoint.

## Vocabulary: CLI aliases vs. manifest values

A routine is a **trigger** (when it fires) plus an **action** (what it does). There are two spellings for each type: the CLI flags accept a short **alias**, while a `--file` manifest (and `azure.yaml`) use the raw **wire `type:` value**. They mean the same thing.

**Triggers**

| Fires on | `--trigger` alias | manifest `type:` | Key fields |
|----------|-------------------|------------------|------------|
| A single moment (one-shot) | `timer` | `timer` | `at` (ISO 8601 UTC) |
| A recurring cron schedule | `recurring` | `schedule` | `cron_expression`, `time_zone` |
| A GitHub issue event | `github-issue` | `github_issue` | `connection_id`, `owner`, `repository`, `issue_event` |
| A custom external event | `custom` | `custom` | `provider`, `event_name`, `parameters` |

**Actions** — both invoke the target agent; they differ only in which agent protocol is called and which field resumes prior context.

| Invokes the agent using | `--action` alias | manifest `type:` | Resume field |
|-------------------------|------------------|------------------|--------------|
| the agent `responses` protocol | `agent-response` (default) | `invoke_agent_responses_api` | `conversation` |
| the agent `invocations` protocol | `agent-invoke` | `invoke_agent_invocations_api` | `session_id` |

## Create

Put the prompt or payload the routine sends to the agent in `action.input`. What it should contain depends on the action type you chose (the `--action` alias / action `type:` from the table above): when the action is `agent-response` (`invoke_agent_responses_api`), `action.input` is the natural-language prompt; when the action is `agent-invoke` (`invoke_agent_invocations_api`), it is the hosted agent's expected request payload. `azd ai routine create` has **no `--input` flag**, so any routine that needs `action.input` must be created from a manifest:

```yaml
# routine.yaml — the type: fields take the manifest value from the table above
triggers:
  default:
    type: schedule
    cron_expression: "0 * * * *"
action:
  type: invoke_agent_responses_api
  agent_name: my-agent
  input: "Say hi."
```

```bash
azd ai routine create hourly-hello --file routine.yaml
```

Flag-only create works only when the target agent needs no stored input. `--file` and `--trigger` are mutually exclusive.

```bash
# One-shot timer -> agent
azd ai routine create nightly-report \
  --trigger timer --at <YYYY-MM-DDTHH:MM:SSZ> \
  --action agent-response --agent-name my-agent

# Recurring cron schedule
azd ai routine create daily-digest \
  --trigger recurring --cron "0 8 * * *" --time-zone America/New_York \
  --action agent-response --agent-name my-agent \
  --description "Daily 8am digest"

# GitHub issue event -> agent
azd ai routine create triage-on-open \
  --trigger github-issue \
  --connection-id <workspace-connection-id> --owner Azure --repository azure-dev \
  --issue-event opened \
  --action agent-invoke --agent-name triage-agent

# Custom event -> agent
azd ai routine create on-custom-event \
  --trigger custom --provider <provider-id> --event-name <event> \
  --parameters '{"key":"value"}' \
  --action agent-response --agent-name my-agent
```

## Create Flags

| Flag | Applies to | Notes |
|------|------------|-------|
| `--trigger` | all | `timer` \| `recurring` \| `github-issue` \| `custom` (required unless `--file`) |
| `--at` | timer | ISO 8601 UTC datetime, e.g. `<YYYY-MM-DDTHH:MM:SSZ>` |
| `--cron` | recurring | 5-field cron; minimum interval 5 minutes |
| `--time-zone` | recurring | IANA zone, e.g. `America/New_York` (default `UTC`; not valid for timer) |
| `--connection-id`, `--owner`, `--repository`, `--issue-event` | github-issue | all four required; `--issue-event` is `opened` or `closed` |
| `--provider`, `--event-name`, `--parameters` | custom | `--provider` and JSON-object `--parameters` required |
| `--action` | all | `agent-response` (default) \| `agent-invoke` |
| `--agent-name` \| `--agent-endpoint-id` | action | exactly one; identifies the target agent |
| `--conversation-id` | agent-response | continue an existing conversation (preview) |
| `--session-id` | agent-invoke | continue an existing hosted-agent session |
| `--description` | all | free-text description |
| `--enabled` | all | enabled by default; pass `--enabled=false` to create disabled |
| `--force` | all | overwrite an existing routine of the same name (upsert) |

## Read

```bash
azd ai routine list
azd ai routine list --output json

azd ai routine show nightly-report
azd ai routine show nightly-report --output json
```

## Update

`update` changes only the fields you pass; everything else is preserved. Supply named flags and/or a `--file` manifest.

```bash
azd ai routine update daily-digest --cron "30 9 * * *"
azd ai routine update daily-digest --agent-name another-agent --description "New owner"
azd ai routine update daily-digest --file routine.yaml
```

The trigger and action **types** are immutable: `--trigger` / `--action` are rejected on `update`. To change a type, delete the routine and recreate it.

## Delete

```bash
azd ai routine delete daily-digest
azd ai routine delete daily-digest --force
```

Use `--force` for non-interactive deletes, including under `--no-prompt`.

## Routine Operations

```bash
azd ai routine enable daily-digest
azd ai routine disable daily-digest

# Fire a routine once, now
azd ai routine dispatch daily-digest
azd ai routine dispatch daily-digest --input '{"foo":"bar"}'
azd ai routine dispatch daily-digest --async

# Inspect past runs
azd ai routine run list daily-digest
azd ai routine run list daily-digest --top 20 --filter "<odata-filter>"
```

`dispatch --input` is a one-time override for that manual run only; it does not change the routine's stored `action.input`. `dispatch` prints a Dispatch ID and Action Correlation ID — use `run list` to see the resulting status and phase.

# Manage Foundry Routines (azd ai routine)

Create, read, update, and delete Microsoft Foundry **routines** with the Azure Developer CLI (`azd`). A routine pairs a trigger (timer, recurring schedule, GitHub issue, or custom external event) with an action that invokes a Foundry agent. Use only the `azd` path for routine work: imperative `azd ai routine` commands or declarative `host: azure.ai.routine` services in `azure.yaml`.

> **Preview.** Routines ship in the `azure.ai.routines` azd extension. The command surface is `azd ai routine <verb>`; do not use Foundry MCP tools, REST, or SDK for routine CRUD in this skill.

## Quick Reference

| Property | Value |
|----------|-------|
| Primary CLI | `azd ai routine` (extension `azure.ai.routines`) |
| Install extension | `azd extension install azure.ai.routines` |
| CRUD verbs | `create`, `list`, `show`, `update`, `delete` |
| Routine operations | `enable`, `disable`, `dispatch`, `run list` |
| Declarative form | `azure.yaml` service with `host: azure.ai.routine`, upserted by `azd deploy` / `azd up` |
| Agent prompt/input | Set `action.input` in a routine manifest or `azure.yaml`; use a string for the agent `responses` protocol and the target payload for the agent `invocations` protocol. `azd ai routine create` flags do not include `--input` |
| Project endpoint | `--project-endpoint`, then `AZURE_AI_PROJECT_ENDPOINT`, global `azd ai project set`, then `FOUNDRY_PROJECT_ENDPOINT` |
| Output format | `--output json` or `--output table` (default) |

## When to Use This Skill

- Schedule an agent on a one-shot timer or recurring cron schedule.
- Trigger an agent from a GitHub issue event or custom external event.
- List, inspect, update, enable, disable, dispatch, or delete existing routines.
- Manage routines declaratively in `azure.yaml` so `azd up` / `azd deploy` keeps them in sync.

> A routine **references an agent**; it does not create one. Deploy or identify the target agent first (see [deploy](../deploy/deploy.md) / [create](../create/create-hosted.md)), then attach a routine to it.

## Workflow

### Step 1 - Verify the environment

Before any routine command, run the shared verification script to confirm `azd`, `az`, auth, and the base Foundry extensions are ready:

```bash
../create/scripts/verify-environment.sh     # macOS / Linux
../create/scripts/verify-environment.ps1    # Windows (pwsh)
```

Act on the summary prefixes:

- `[OK]` - nothing to do.
- `[WARN]` - non-blocking; continue.
- `[ACTION]` - resolve first, then rerun the script. Never run `az login` or `azd auth login` for the user; stop and ask them to log in manually. Missing base extensions (`azure.ai.agents`, `azure.ai.projects`, `microsoft.foundry`) can be installed with `azd extension install <name>`.

Do not continue while any `[ACTION]` remains.

### Step 1b - Check the routines extension

The shared script does not check `azure.ai.routines`. Confirm it is installed:

```bash
azd extension list --installed --output json
```

If missing, install it (ask first in interactive mode; install directly in non-interactive mode):

```bash
azd extension install azure.ai.routines
```

Verify the command surface:

```bash
azd ai routine --help
```

If `azd ai routine` reports an unknown command after install, the azd core is too old. The extension requires `azd >= 1.27.0`; upgrade azd (<https://aka.ms/azd-install>) and retry.

### Step 2 - Resolve the Foundry project endpoint

Every routine command targets a Foundry project endpoint. `azd ai routine` resolves it in this order:

1. `-p` / `--project-endpoint <url>` on the command.
2. Active azd environment `AZURE_AI_PROJECT_ENDPOINT` (`azd env get-values`).
3. Global config from `azd ai project set <endpoint>`.
4. `FOUNDRY_PROJECT_ENDPOINT` environment variable.
5. Otherwise the command fails with a missing-endpoint error.

Prefer the azd env inside an azd project. Otherwise set it once:

```bash
azd env set AZURE_AI_PROJECT_ENDPOINT "https://<account>.services.ai.azure.com/api/projects/<project>"
# or, outside an azd project:
azd ai project set "https://<account>.services.ai.azure.com/api/projects/<project>"
```

The endpoint host must end with `.services.ai.azure.com` and use `https` with no explicit port.

## Two Ways to Create a Routine

A routine is the same Foundry resource — keyed by its name — no matter how you create it. Both paths go through `azd` and act on that same named resource, so a routine created one way can later be managed the other way. Declarative `azd deploy` always upserts idempotently; imperative `azd ai routine create` refuses to overwrite an existing routine unless you pass `--force`. Pick a path, then read its reference doc for exact examples.

### Way 1 — Imperative: `azd ai routine create`

Create the routine directly against the Foundry project with a single command — flags, or a `--file` manifest when it must carry a stored prompt/payload (`action.input`; there is no `--input` flag). Best for one-off scheduling, quick experiments, ad-hoc CRUD, and working **outside** an azd project — no `azure.yaml` required. → [CLI CRUD and Operations](references/cli-crud.md)

### Way 2 — Declarative: `azure.yaml` + `azd deploy`

Declare the routine as a `host: azure.ai.routine` service in `azure.yaml`, then let `azd deploy` / `azd up` upsert it. Best when the routine should be **versioned with the agent in source control** and **reproduced per azd environment** — GitOps, multi-env, CI/CD. → [Declarative Routines](references/azure-yaml.md)

### Which path?

| Situation | Path |
|-----------|------|
| One-off schedule, quick experiment, or no `azure.yaml` in play | Way 1 — imperative |
| Routine versioned with the agent, reproduced per environment, GitOps / CI/CD | Way 2 — declarative |
| Unsure and already in an azd project with the agent | Way 2 — declarative keeps the routine and agent in sync |

Read, update, enable/disable, manually dispatch, inspect past runs, and delete are imperative-only operations that work on a routine regardless of how it was created — see [CLI CRUD and Operations](references/cli-crud.md).

## Error Handling

| Symptom | Cause | Resolution |
|---------|-------|------------|
| `unknown command "routine"` / `unknown command "ai"` | Extension not installed or azd too old | `azd extension install azure.ai.routines`; ensure `azd >= 1.27.0` |
| Missing project endpoint error | No endpoint resolved | Set `AZURE_AI_PROJECT_ENDPOINT`, run `azd ai project set <url>`, or pass `-p <url>` |
| `routine "<name>" already exists` on create | Name collision | Re-run with `--force` to upsert, or choose a different name |
| `--trigger cannot be changed on an existing routine` (same for `--action`) | Trigger/action type is immutable | Delete then create with the new type |
| `--force is required when --no-prompt is set` on delete | Non-interactive delete without confirmation | Add `--force` |
| `routine "<name>" not found` | Wrong name or wrong project | Check the name and resolved endpoint with `show` / `list` |
| `host "..." is not a recognized Foundry host` | Endpoint host invalid | Use `https://<account>.services.ai.azure.com/api/projects/<project>` (no port) |
| `json: cannot unmarshal number into Go struct field Routine.created_at of type string` | The routines extension could not decode a routine response after the service call | Do not assume the operation failed. Check with `show <name>` and `list`; if both decode badly, the routine may exist but cannot be decoded by the current extension. |
| Network isolation / `PublicNetworkAccessDisabled` / `403` | Project has public access disabled | See [Network Isolation Errors](../../SKILL.md#network-isolation-errors) |

## Additional Resources

- [CLI CRUD and Operations](references/cli-crud.md)
- [Declarative Routines](references/azure-yaml.md)
- [azd ai CLI Reference](../create/references/azd-ai-cli.md)
- [Deploy a Foundry Agent](../deploy/deploy.md) - deploy the agent a routine will invoke
- [Invoke a Foundry Agent](../invoke/invoke.md) - smoke-test the agent before scheduling it
- [Microsoft Foundry Skill (index)](../../SKILL.md)

# Analyze Failures — Find and Cluster Failing Traces

Identify failing agent traces, group them by root cause, and produce a prioritized action table.

## Step 1 — Find Failing Traces

> ⚠️ **Hosted agents:** `gen_ai.agent.name` on `dependencies` holds the **code-level class name** (e.g., `BingSearchAgent`), NOT the Foundry agent name. To filter by Foundry name, use the [Hosted Agent Variant](#hosted-agent-variant--failures) below.

```kql
dependencies
| where timestamp > ago(24h)
| where success == false or toint(resultCode) >= 400
| extend
    operation = tostring(customDimensions["gen_ai.operation.name"]),
    errorType = tostring(customDimensions["error.type"]),
    model = tostring(customDimensions["gen_ai.request.model"]),
    agentName = tostring(customDimensions["gen_ai.agent.name"]),
    conversationId = tostring(customDimensions["gen_ai.conversation.id"])
| project timestamp, name, duration, resultCode, errorType, operation, model,
    agentName, conversationId, operation_Id, id
| order by timestamp desc
| take 100
```

## Step 2 — Cluster by Error Type

```kql
dependencies
| where timestamp > ago(24h)
| where success == false or toint(resultCode) >= 400
| extend
    errorType = tostring(customDimensions["error.type"]),
    operation = tostring(customDimensions["gen_ai.operation.name"])
| summarize
    count = count(),
    firstSeen = min(timestamp),
    lastSeen = max(timestamp),
    avgDuration = avg(duration),
    sampleOperationId = take_any(operation_Id)
  by errorType, operation, resultCode
| order by count desc
```

## Step 3 — Prioritized Action Table

Present results as:

| Priority | Error Type | Operation | Count | Result Code | Suggested Action |
|----------|-----------|-----------|-------|-------------|-----------------|
| P0 | timeout | invoke_agent | 15 | 504 | Check agent container health, increase timeout |
| P1 | rate_limited | chat | 8 | 429 | Check quota, add retry logic |
| P2 | content_filter | chat | 5 | 400 | Review prompt for policy violations |
| P3 | tool_error | execute_tool | 3 | 500 | Check tool implementation and permissions |

**Prioritization:** P0 = highest count or most severe (5xx), then by count × recency.

## Step 4 — Drill Into Specific Failure

When the user selects a cluster, show individual failing traces:

```kql
dependencies
| where timestamp > ago(24h)
| where success == false
| where customDimensions["error.type"] == "<selected_error_type>"
| where customDimensions["gen_ai.operation.name"] == "<selected_operation>"
| project timestamp, name, duration, resultCode,
    conversationId = tostring(customDimensions["gen_ai.conversation.id"]),
    responseId = tostring(customDimensions["gen_ai.response.id"]),
    operation_Id
| order by timestamp desc
| take 20
```

Also check `exceptions` table for stack traces:

```kql
exceptions
| where timestamp > ago(24h)
| where operation_Id in ("<operation_id_1>", "<operation_id_2>")
| project timestamp, type, message, outerMessage, details, operation_Id
| order by timestamp desc
```

Offer to view the full conversation for any trace via [Conversation Detail](conversation-detail.md).

## Hosted Agent Variant — Failures

For hosted agents, the Foundry agent name lives on `requests`, not `dependencies`. Use a two-step join:

```kql
let reqIds = requests
| where timestamp > ago(24h)
| where customDimensions["gen_ai.agent.name"] == "<foundry-agent-name>"
| distinct id;
dependencies
| where timestamp > ago(24h)
| where operation_ParentId in (reqIds)
| where success == false or toint(resultCode) >= 400
| extend
    operation = tostring(customDimensions["gen_ai.operation.name"]),
    errorType = tostring(customDimensions["error.type"]),
    model = tostring(customDimensions["gen_ai.request.model"]),
    conversationId = tostring(customDimensions["gen_ai.conversation.id"])
| project timestamp, name, duration, resultCode, errorType, operation, model,
    conversationId, operation_ParentId, operation_Id
| order by timestamp desc
| take 100
```

# Analyze Latency — Find and Diagnose Slow Traces

Identify slow agent traces, find bottleneck spans, and correlate with token usage.

## Step 1 — Find Slow Conversations

> ⚠️ **Hosted agents:** `gen_ai.agent.name` on `dependencies` holds the **code-level class name** (e.g., `BingSearchAgent`), NOT the Foundry agent name. To scope by Foundry name, use the [Hosted Agent Variant](#hosted-agent-variant--latency) below.

```kql
dependencies
| where timestamp > ago(24h)
| where customDimensions["gen_ai.operation.name"] == "invoke_agent"
| project timestamp, duration, success,
    agentName = tostring(customDimensions["gen_ai.agent.name"]),
    conversationId = tostring(customDimensions["gen_ai.conversation.id"]),
    operation_Id
| summarize
    totalDuration = sum(duration),
    spanCount = count(),
    hasErrors = countif(success == false) > 0
  by conversationId, operation_Id
| where totalDuration > 5000
| order by totalDuration desc
| take 50
```

> **Default threshold:** 5 seconds. Ask the user for their latency threshold if not specified.

## Step 2 — Latency Distribution (P50/P95/P99)

```kql
dependencies
| where timestamp > ago(24h)
| where customDimensions["gen_ai.operation.name"] in ("chat", "invoke_agent")
| summarize
    p50 = percentile(duration, 50),
    p95 = percentile(duration, 95),
    p99 = percentile(duration, 99),
    avg = avg(duration),
    count = count()
  by operation = tostring(customDimensions["gen_ai.operation.name"]),
     model = tostring(customDimensions["gen_ai.request.model"])
| order by p95 desc
```

Present as:

| Operation | Model | P50 (ms) | P95 (ms) | P99 (ms) | Avg (ms) | Count |
|-----------|-------|---------|---------|---------|---------|-------|

## Step 3 — Bottleneck Breakdown

For a specific slow conversation, break down time spent per span type:

```kql
dependencies
| where operation_Id == "<operation_id>"
| extend operation = tostring(customDimensions["gen_ai.operation.name"])
| summarize
    totalDuration = sum(duration),
    spanCount = count(),
    avgDuration = avg(duration)
  by operation, name
| order by totalDuration desc
```

Common bottleneck patterns:
- **`chat` spans dominate** → LLM inference is slow (consider smaller model or caching)
- **`execute_tool` spans dominate** → Tool execution is slow (optimize tool implementation)
- **`invoke_agent` has long gaps** → Orchestration overhead (check agent framework)

## Step 4 — Token Usage vs Latency Correlation

```kql
dependencies
| where timestamp > ago(24h)
| where customDimensions["gen_ai.operation.name"] == "chat"
| extend
    inputTokens = toint(customDimensions["gen_ai.usage.input_tokens"]),
    outputTokens = toint(customDimensions["gen_ai.usage.output_tokens"])
| where isnotempty(inputTokens)
| project duration, inputTokens, outputTokens,
    model = tostring(customDimensions["gen_ai.request.model"]),
    operation_Id
| order by duration desc
| take 100
```

High token counts often correlate with high latency. If confirmed, suggest:
- Reduce system prompt length
- Limit conversation history window
- Use a faster model for simpler queries

## Hosted Agent Variant — Latency

For hosted agents, scope by Foundry agent name via `requests` then join to `dependencies`:

```kql
let reqIds = requests
| where timestamp > ago(24h)
| where customDimensions["gen_ai.agent.name"] == "<foundry-agent-name>"
| distinct id;
dependencies
| where timestamp > ago(24h)
| where operation_ParentId in (reqIds)
| where customDimensions["gen_ai.operation.name"] in ("chat", "invoke_agent")
| summarize
    p50 = percentile(duration, 50),
    p95 = percentile(duration, 95),
    p99 = percentile(duration, 99),
    avg = avg(duration),
    count = count()
  by operation = tostring(customDimensions["gen_ai.operation.name"]),
     model = tostring(customDimensions["gen_ai.request.model"])
| order by p95 desc
```

# Conversation Detail — Reconstruct Full Span Tree

Reconstruct the complete span tree for a single conversation to see exactly what happened: every LLM call, tool execution, and agent invocation with timing, tokens, and errors.

## Step 1 — Fetch All Spans for a Conversation

Use `operation_Id` (trace ID) to get all spans in a single request:

```kql
dependencies
| where operation_Id == "<operation_id>"
| project timestamp, name, duration, resultCode, success,
    spanId = id,
    parentSpanId = operation_ParentId,
    operation = tostring(customDimensions["gen_ai.operation.name"]),
    model = tostring(customDimensions["gen_ai.request.model"]),
    responseModel = tostring(customDimensions["gen_ai.response.model"]),
    inputTokens = toint(customDimensions["gen_ai.usage.input_tokens"]),
    outputTokens = toint(customDimensions["gen_ai.usage.output_tokens"]),
    responseId = tostring(customDimensions["gen_ai.response.id"]),
    finishReason = tostring(customDimensions["gen_ai.response.finish_reasons"]),
    errorType = tostring(customDimensions["error.type"]),
    toolName = tostring(customDimensions["gen_ai.tool.name"]),
    toolCallId = tostring(customDimensions["gen_ai.tool.call.id"])
| order by timestamp asc
```

Also fetch the parent request:

```kql
requests
| where operation_Id == "<operation_id>"
| project timestamp, name, duration, resultCode, success, id, operation_ParentId
```

## Step 2 — Build Span Tree

Use `spanId` and `parentSpanId` to reconstruct the hierarchy:

```
invoke_agent (root) ─── 4200ms
├── chat (LLM call #1) ─── 1800ms, gpt-4o, 450→120 tokens
│   └── [output: "Let me check the weather..."]
├── execute_tool (get_weather) [tool: remote_functions.weather_api] ─── 200ms
│   └── [result: "rainy, 57°F"]
├── chat (LLM call #2) ─── 1500ms, gpt-4o, 620→85 tokens
│   └── [output: "The weather in Paris is rainy, 57°F"]
└── [total: 450+620=1070 input, 120+85=205 output tokens]
```

Present as an indented tree with:
- **Operation type** and name
- **Duration** (highlight if > P95 for that operation type)
- **Model** and token counts (for chat operations)
- **Error type** and result code (if failed, highlight in red)
- **Finish reason** (stop, length, content_filter, tool_calls)

## Step 3 — Extract Conversation Content from invoke_agent Spans

The full input/output content lives on `invoke_agent` dependency spans in `gen_ai.input.messages` and `gen_ai.output.messages`. These JSON arrays contain the complete conversation (system prompt, user query, assistant response):

```kql
dependencies
| where operation_Id == "<operation_id>"
| where customDimensions["gen_ai.operation.name"] == "invoke_agent"
| project timestamp,
    inputMessages = tostring(customDimensions["gen_ai.input.messages"]),
    outputMessages = tostring(customDimensions["gen_ai.output.messages"])
| order by timestamp asc
```

Message structure: `[{"role": "user", "parts": [{"type": "text", "content": "..."}]}]`

Also check the `traces` table for additional GenAI log events:

```kql
traces
| where operation_Id == "<operation_id>"
| where message contains "gen_ai"
| project timestamp, message, customDimensions
| order by timestamp asc
```

## Step 4 — Check for Exceptions

```kql
exceptions
| where operation_Id == "<operation_id>"
| project timestamp, type, message, outerMessage,
    details = parse_json(details)
| order by timestamp asc
```

Present exceptions inline in the span tree at their position in the timeline.

## Step 5 — Fetch Evaluation Results

See [Eval Correlation](eval-correlation.md) for the full workflow to look up evaluation scores by response ID or conversation ID. Use `gen_ai.response.id` values from Step 1 spans to correlate.

# Eval Correlation — Find Evaluation Results by Response or Conversation ID

Look up evaluation scores for a specific agent response using App Insights.

> **IMPORTANT:** The Foundry evaluation API does NOT support querying by response ID or conversation ID. App Insights `customEvents` is the ONLY way to correlate eval scores to specific responses. Always use this KQL approach when the user asks for eval results for a specific response or conversation.

## Prerequisites

- App Insights resource resolved (see [trace.md](../trace.md) Before Starting)
- A response ID (`gen_ai.response.id`) or conversation ID (`gen_ai.conversation.id`) from a previous trace query

## Search by Response ID

```kql
customEvents
| where timestamp > ago(30d)
| where name == "gen_ai.evaluation.result"
| where customDimensions["gen_ai.response.id"] == "<response_id>"
| extend
    evalName = tostring(customDimensions["gen_ai.evaluation.name"]),
    score = todouble(customDimensions["gen_ai.evaluation.score.value"]),
    label = tostring(customDimensions["gen_ai.evaluation.score.label"]),
    explanation = tostring(customDimensions["gen_ai.evaluation.explanation"]),
    responseId = tostring(customDimensions["gen_ai.response.id"]),
    conversationId = tostring(customDimensions["gen_ai.conversation.id"])
| project timestamp, evalName, score, label, explanation, responseId, conversationId
| order by evalName asc
```

## Search by Conversation ID

```kql
customEvents
| where timestamp > ago(30d)
| where name == "gen_ai.evaluation.result"
| where customDimensions["gen_ai.conversation.id"] == "<conversation_id>"
| extend
    evalName = tostring(customDimensions["gen_ai.evaluation.name"]),
    score = todouble(customDimensions["gen_ai.evaluation.score.value"]),
    label = tostring(customDimensions["gen_ai.evaluation.score.label"]),
    explanation = tostring(customDimensions["gen_ai.evaluation.explanation"]),
    responseId = tostring(customDimensions["gen_ai.response.id"])
| project timestamp, evalName, score, label, explanation, responseId
| order by responseId asc, evalName asc
```

## Present Results

Show eval scores as a table:

| Evaluator | Score | Label | Explanation |
|-----------|-------|-------|-------------|
| coherence | 5.0 | pass | Response is well-structured... |
| fluency | 4.0 | pass | Natural language flow... |
| relevance | 2.0 | fail | Response doesn't address... |

When showing alongside a span tree (see [Conversation Detail](conversation-detail.md)), attach eval scores to the span whose `gen_ai.response.id` matches.

# KQL Templates — GenAI Trace Query Reference

Ready-to-use KQL templates for querying GenAI OpenTelemetry traces in Application Insights.

**Table of Contents:** [App Insights Table Mapping](#app-insights-table-mapping) · [Key GenAI OTel Attributes](#key-genai-otel-attributes) · [Span Correlation](#span-correlation) · [Hosted Agent Attributes](#hosted-agent-attributes) · [Response ID Formats](#response-id-formats) · [Common Query Templates](#common-query-templates) · [OTel Reference Links](#otel-reference-links)

## App Insights Table Mapping

| App Insights Table | GenAI Data |
|-------------------|------------|
| `dependencies` | GenAI spans: LLM inference (`chat`), tool execution (`execute_tool`), agent invocation (`invoke_agent`) |
| `requests` | Incoming HTTP requests to the agent endpoint. For hosted agents, also carries `gen_ai.agent.name` (Foundry name) and `azure.ai.agentserver.*` attributes — **preferred entry point** for agent-name filtering |
| `customEvents` | GenAI evaluation results (`gen_ai.evaluation.result`) — scores, labels, explanations |
| `traces` | Log events, including GenAI events (input/output messages) |
| `exceptions` | Error details with stack traces |

## Key GenAI OTel Attributes

Stored in `customDimensions` on `dependencies` spans:

| Attribute | Description | Example |
|-----------|-------------|---------|
| `gen_ai.operation.name` | Operation type | `chat`, `invoke_agent`, `execute_tool`, `create_agent` |
| `gen_ai.conversation.id` | Conversation/session ID | `conv_5j66UpCpwteGg4YSxUnt7lPY` |
| `gen_ai.response.id` | Response ID | `chatcmpl-123` |
| `gen_ai.agent.name` | Agent name | `my-support-agent` |
| `gen_ai.agent.id` | Agent identifier | `asst_abc123` |
| `gen_ai.request.model` | Requested model | `gpt-4o` |
| `gen_ai.response.model` | Actual model used | `gpt-4o-2024-05-13` |
| `gen_ai.usage.input_tokens` | Input token count | `450` |
| `gen_ai.usage.output_tokens` | Output token count | `120` |
| `gen_ai.response.finish_reasons` | Stop reasons | `["stop"]`, `["tool_calls"]` |
| `error.type` | Error classification | `timeout`, `rate_limited`, `content_filter` |
| `gen_ai.provider.name` | Provider | `azure.ai.openai`, `openai` |
| `gen_ai.input.messages` | Full input messages (JSON array) — on `invoke_agent` spans | `[{"role":"user","parts":[{"type":"text","content":"..."}]}]` |
| `gen_ai.output.messages` | Full output messages (JSON array) — on `invoke_agent` spans | `[{"role":"assistant","parts":[{"type":"text","content":"..."}]}]` |

Stored in `customDimensions` on `customEvents` (name == `gen_ai.evaluation.result`):

| Attribute | Description | Example |
|-----------|-------------|---------|
| `gen_ai.evaluation.name` | Evaluator name | `Relevance`, `IntentResolution` |
| `gen_ai.evaluation.score.value` | Numeric score | `4.0` |
| `gen_ai.evaluation.score.label` | Human-readable label | `pass`, `fail`, `relevant` |
| `gen_ai.evaluation.explanation` | Free-form explanation | `"Response lacks detail..."` |
| `gen_ai.response.id` | Correlates to the evaluated span | `chatcmpl-123` |
| `gen_ai.conversation.id` | Correlates to conversation | `conv_5j66...` |

> **Correlation:** Eval results do NOT link via id-parentId. Use `gen_ai.conversation.id` and/or `gen_ai.response.id` to join with `dependencies` spans.

## Span Correlation

| Field | Purpose |
|-------|---------|
| `operation_Id` | Trace ID — groups all spans in one request |
| `id` | Span ID — unique identifier for this span |
| `operation_ParentId` | Parent span ID — use with `id` to build span trees |

### Operation_Id Join (requests → dependencies)

Use `requests` as the hosted-agent entry point, then carry `operation_Id` forward as the trace key when joining into `dependencies`, `traces`, or `customEvents`:

```kql
let agentRequests = materialize(
    requests
| where timestamp > ago(7d)
| extend
    foundryAgentName = coalesce(
        tostring(customDimensions["gen_ai.agent.name"]),
        tostring(customDimensions["azure.ai.agentserver.agent_name"])
    ),
    agentId = tostring(customDimensions["gen_ai.agent.id"]),
    agentNameFromId = tostring(split(agentId, ":")[0]),
    agentVersion = iff(agentId contains ":", tostring(split(agentId, ":")[1]), ""),
    conversationId = coalesce(
        tostring(customDimensions["gen_ai.conversation.id"]),
        tostring(customDimensions["azure.ai.agentserver.conversation_id"]),
        operation_Id
    )
| where foundryAgentName == "<foundry-agent-name>"
    or agentNameFromId == "<foundry-agent-name>"
| project operation_Id, conversationId, agentVersion
);
dependencies
| where timestamp > ago(7d)
| where isnotempty(customDimensions["gen_ai.operation.name"])
| join kind=inner agentRequests on operation_Id
| extend
    operation = tostring(customDimensions["gen_ai.operation.name"]),
    model = tostring(customDimensions["gen_ai.request.model"])
| project timestamp, duration, success, operation, model, conversationId, agentVersion, operation_Id
| order by timestamp desc
```

## Hosted Agent Attributes

Stored in `customDimensions` on **both `requests` and `traces`** tables (NOT on `dependencies` spans):

| Attribute | Description | Example |
|-----------|-------------|---------|
| `azure.ai.agentserver.agent_name` | Hosted agent name | `hosted-agent-022-001` |
| `azure.ai.agentserver.agent_id` | Internal agent ID | `code-asst-xmwokux85uqc7fodxejaxa` |
| `azure.ai.agentserver.conversation_id` | Conversation ID | `conv_d7ab624de92d...` |
| `azure.ai.agentserver.response_id` | Response ID (caresp format) | `caresp_d7ab624de92d...` |

> **Important:** Use `requests` as the preferred entry point for agent-name filtering — it has both `azure.ai.agentserver.agent_name` and `gen_ai.agent.name` with the Foundry-level name. To reach downstream spans and related telemetry, carry `operation_Id` forward from the filtered request set and join other tables on that trace key.

> 💡 **Version enrichment:** Some hosted-agent `requests` telemetry emits `gen_ai.agent.id` in `<foundry-agent-name>:<version>` format. When that delimiter is present, split on `:` to recover `agentVersion`; if it is absent, keep filtering on the requests-scoped name fields and leave version blank.

> ⚠️ **`gen_ai.agent.name` means different things on different tables:**
> - On `requests`: the **Foundry agent name** (user-visible) → e.g., `hosted-agent-022-001`
> - On `dependencies`: the **code-level class name** → e.g., `BingSearchAgent`
>
> **Always start from `requests`** when filtering by the Foundry agent name the user knows.

## Response ID Formats

| Agent Type | Prefix | Example |
|------------|--------|---------|
| Hosted agent (AgentServer) | `caresp_` | `caresp_d7ab624de92da637008Rhr4U4E1y9FSE...` |
| Prompt agent (Foundry Responses API) | `resp_` | `resp_4e2f8b016b5a0dad00697bd3c4c1b881...` |
| Azure OpenAI chat completions | `chatcmpl-` | `chatcmpl-abc123def456` |

When searching by response ID, use the appropriate prefix to narrow results. The `gen_ai.response.id` attribute appears on `dependencies` spans (for `chat` operations) and in `customEvents` (for evaluation results).

## Common Query Templates

### Overview — Conversations in last 24h
```kql
dependencies
| where timestamp > ago(24h)
| where isnotempty(customDimensions["gen_ai.operation.name"])
| summarize
    spanCount = count(),
    errorCount = countif(success == false),
    avgDuration = avg(duration),
    totalInputTokens = sum(toint(customDimensions["gen_ai.usage.input_tokens"])),
    totalOutputTokens = sum(toint(customDimensions["gen_ai.usage.output_tokens"]))
  by bin(timestamp, 1h)
| order by timestamp desc
```

### Error Rate by Operation
```kql
dependencies
| where timestamp > ago(24h)
| where isnotempty(customDimensions["gen_ai.operation.name"])
| summarize
    total = count(),
    errors = countif(success == false),
    errorRate = round(100.0 * countif(success == false) / count(), 1)
  by operation = tostring(customDimensions["gen_ai.operation.name"])
| order by errorRate desc
```

### Token Usage by Model
```kql
dependencies
| where timestamp > ago(24h)
| where customDimensions["gen_ai.operation.name"] == "chat"
| summarize
    calls = count(),
    totalInput = sum(toint(customDimensions["gen_ai.usage.input_tokens"])),
    totalOutput = sum(toint(customDimensions["gen_ai.usage.output_tokens"])),
    avgInput = avg(todouble(customDimensions["gen_ai.usage.input_tokens"])),
    avgOutput = avg(todouble(customDimensions["gen_ai.usage.output_tokens"]))
  by model = tostring(customDimensions["gen_ai.request.model"])
| order by totalInput desc
```

### Tool Call Details
```kql
dependencies
| where operation_Id == "<operation_id>"
| where customDimensions["gen_ai.operation.name"] == "execute_tool"
| project timestamp, duration, success,
    toolName = tostring(customDimensions["gen_ai.tool.name"]),
    toolType = tostring(customDimensions["gen_ai.tool.type"]),
    toolCallId = tostring(customDimensions["gen_ai.tool.call.id"]),
    toolArgs = tostring(customDimensions["gen_ai.tool.call.arguments"]),
    toolResult = tostring(customDimensions["gen_ai.tool.call.result"])
| order by timestamp asc
```

Key tool attributes:

| Attribute | Description | Example |
|-----------|-------------|---------|
| `gen_ai.tool.name` | Tool function name | `remote_functions.bing_grounding`, `python` |
| `gen_ai.tool.type` | Tool type | `extension`, `function` |
| `gen_ai.tool.call.id` | Unique call ID | `call_db64aa6a004a...` |
| `gen_ai.tool.call.arguments` | JSON arguments passed | `{"query": "latest AI news"}` |
| `gen_ai.tool.call.result` | Tool output (may be truncated) | `<<ImageDisplayed>>` |

### Evaluation Results by Conversation
```kql
customEvents
| where timestamp > ago(24h)
| where name == "gen_ai.evaluation.result"
| extend
    evalName = tostring(customDimensions["gen_ai.evaluation.name"]),
    score = todouble(customDimensions["gen_ai.evaluation.score.value"]),
    label = tostring(customDimensions["gen_ai.evaluation.score.label"]),
    conversationId = tostring(customDimensions["gen_ai.conversation.id"])
| summarize
    evalCount = count(),
    avgScore = avg(score),
    failCount = countif(label == "fail" or label == "not_relevant" or label == "incorrect"),
    evaluators = make_set(evalName)
  by conversationId
| order by failCount desc
```

> For detailed eval queries by response ID or conversation ID, see [Eval Correlation](eval-correlation.md).

## OTel Reference Links

- [GenAI Spans](https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-spans/)
- [GenAI Agent Spans](https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-agent-spans/)
- [GenAI Events](https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-events/)
- [GenAI Metrics](https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-metrics/)

# Search Traces — Conversation-Level Search

Search agent traces at the conversation level. Returns summaries grouped by conversation or operation, not individual spans.

## Prerequisites

- App Insights resource resolved (see [trace.md](../trace.md) Before Starting)
- Selected agent root, environment, effective context source, and metadata overlay confirmed
- Time range confirmed with user (default: last 24 hours)

## Search by Conversation ID

Keep the selected environment visible in the summary, and add the selected agent name or environment tag filters when the telemetry emits them.

```kql
dependencies
| where timestamp > ago(24h)
| where customDimensions["gen_ai.conversation.id"] == "<conversation_id>"
| project timestamp, name, duration, resultCode, success,
    operation = tostring(customDimensions["gen_ai.operation.name"]),
    model = tostring(customDimensions["gen_ai.request.model"]),
    inputTokens = toint(customDimensions["gen_ai.usage.input_tokens"]),
    outputTokens = toint(customDimensions["gen_ai.usage.output_tokens"]),
    operation_Id, id, operation_ParentId
| order by timestamp asc
```

## Search by Response ID

Auto-detect the response ID format to determine agent type:
- `caresp_...` → Hosted agent (AgentServer)
- `resp_...` → Prompt agent (Foundry Responses API)
- `chatcmpl-...` → Azure OpenAI chat completions

```kql
dependencies
| where timestamp > ago(24h)
| where customDimensions["gen_ai.response.id"] == "<response_id>"
| project timestamp, name, duration, resultCode, success,
    operation = tostring(customDimensions["gen_ai.operation.name"]),
    model = tostring(customDimensions["gen_ai.request.model"]),
    inputTokens = toint(customDimensions["gen_ai.usage.input_tokens"]),
    outputTokens = toint(customDimensions["gen_ai.usage.output_tokens"]),
    operation_Id, id, operation_ParentId
```

Then drill into the full conversation:

> ⚠️ **STOP — read [Conversation Detail](conversation-detail.md) before writing your own drill-down query.** It contains the correct span tree reconstruction logic, event/exception queries, and eval correlation steps.

Quick drill-down using the `operation_Id` from above:

```kql
dependencies
| where operation_Id == "<operation_id_from_above>"
| project timestamp, name, duration, resultCode, success,
    spanId = id, parentSpanId = operation_ParentId,
    operation = tostring(customDimensions["gen_ai.operation.name"]),
    model = tostring(customDimensions["gen_ai.request.model"]),
    inputTokens = toint(customDimensions["gen_ai.usage.input_tokens"]),
    outputTokens = toint(customDimensions["gen_ai.usage.output_tokens"]),
    responseId = tostring(customDimensions["gen_ai.response.id"]),
    errorType = tostring(customDimensions["error.type"]),
    toolName = tostring(customDimensions["gen_ai.tool.name"])
| order by timestamp asc
```

Also check for eval results: see [Eval Correlation](eval-correlation.md).

## Search by Agent Name

> **Note:** For hosted agents, `gen_ai.agent.name` in `dependencies` refers to *sub-agents* (e.g., `BingSearchAgent`), not the top-level hosted agent. See "Search by Hosted Agent Name" below.

> 💡 **Hosted-agent versioning:** If you need the deployed version, use the hosted-agent pattern below and parse `gen_ai.agent.id` when it is emitted in `<agent-name>:<version>` format.

```kql
dependencies
| where timestamp > ago(24h)
| where customDimensions["gen_ai.agent.name"] == "<agent_name>"
| summarize
    startTime = min(timestamp),
    endTime = max(timestamp),
    totalDuration = max(timestamp) - min(timestamp),
    spanCount = count(),
    errorCount = countif(success == false),
    totalInputTokens = sum(toint(customDimensions["gen_ai.usage.input_tokens"])),
    totalOutputTokens = sum(toint(customDimensions["gen_ai.usage.output_tokens"]))
  by conversationId = tostring(customDimensions["gen_ai.conversation.id"]),
     operation_Id
| order by startTime desc
| take 50
```

## Search by Hosted Agent Name

For hosted agents, the Foundry agent name (e.g., `hosted-agent-022-001`) appears on `requests` and `traces` — NOT on `dependencies`. Use `requests` as the preferred entry point, materialize the matching request rows, then join downstream spans on `operation_Id`:

```kql
let agentRequests = materialize(
    requests
| where timestamp > ago(24h)
| extend
    foundryAgentName = coalesce(
        tostring(customDimensions["gen_ai.agent.name"]),
        tostring(customDimensions["azure.ai.agentserver.agent_name"])
    ),
    agentId = tostring(customDimensions["gen_ai.agent.id"]),
    agentNameFromId = tostring(split(agentId, ":")[0]),
    agentVersion = iff(agentId contains ":", tostring(split(agentId, ":")[1]), ""),
    conversationId = coalesce(
        tostring(customDimensions["gen_ai.conversation.id"]),
        tostring(customDimensions["azure.ai.agentserver.conversation_id"]),
        operation_Id
    )
| where foundryAgentName == "<agent_name>"
    or agentNameFromId == "<agent_name>"
| project operation_Id, conversationId, agentVersion
);
dependencies
| where timestamp > ago(24h)
| where isnotempty(customDimensions["gen_ai.operation.name"])
| join kind=inner agentRequests on operation_Id
| summarize
    startTime = min(timestamp),
    endTime = max(timestamp),
    spanCount = count(),
    errorCount = countif(success == false),
    totalInputTokens = sum(toint(customDimensions["gen_ai.usage.input_tokens"])),
    totalOutputTokens = sum(toint(customDimensions["gen_ai.usage.output_tokens"]))
  by conversationId, operation_Id, agentVersion
| order by startTime desc
| take 50
```

If `gen_ai.agent.id` does not contain `:`, continue using the requests-scoped name fields for filtering and treat `agentVersion` as optional enrichment rather than a required key.

## Conversation Summary Table

Present results in this format:

| Conversation ID | Agent Version | Start Time | Duration | Spans | Errors | Input Tokens | Output Tokens |
|----------------|---------------|------------|----------|-------|--------|-------------|---------------|
| conv_abc123 | 3 | 2025-01-15 10:30 | 4.2s | 12 | 0 | 850 | 320 |
| conv_def456 | 4 | 2025-01-15 10:25 | 8.7s | 18 | 2 | 1200 | 450 |

Highlight rows with errors in the summary. Offer to drill into any conversation via [Conversation Detail](conversation-detail.md).

## Free-Text Search

When the user provides a general search term (e.g., agent name, error message):

```kql
union dependencies, requests, exceptions, traces
| where timestamp > ago(24h)
| where * contains "<search_term>"
| summarize count() by operation_Id
| order by count_ desc
| take 20
```

## After Successful Query

> 📝 **Reminder:** If this is the first trace query in this session, ensure App Insights connection info was persisted to the selected metadata file for the selected environment (see [trace.md — Before Starting](../trace.md#before-starting--resolve-app-insights-connection)).

# Tracing Insights API

Automatically detect quality regressions and anomalies in agent traces using changepoint detection on evaluation scores stored in App Insights.

## When to Use

Use this instead of manual KQL queries when you want **automated anomaly detection** across evaluation dimensions (task adherence, intent resolution, fluency, latency, token usage). The API finds statistical changepoints in score distributions — no manual threshold tuning needed.

**Prerequisites:**
- App Insights connected to the Foundry project (with `gen_ai.evaluation.result` custom events)
- Evaluation data from portal playground sessions or batch evals (raw traces alone are not enough)

## Endpoint

```
POST https://{region}.api.azureml.ms/notification/v1-beta2/subscriptions/{sub}/resourceGroups/{rg}/providers/microsoft.insights/components/{component}/:insights
```

The API is region-agnostic — any regional endpoint can serve requests for any project. For lowest latency, use the same region as the Foundry project (e.g., `eastus2`, `westus2`, `westcentralus`). If the project region is unknown, use `eastus2` as the default.

**Query parameters:**
| Parameter          | Required | Description                                                            |
|--------------------|----------|------------------------------------------------------------------------|
| `startDateTimeUtc` | Yes      | ISO 8601 start of analysis window                                      |
| `endDateTimeUtc`   | Yes      | ISO 8601 end of analysis window                                        |
| `agent`            | Yes      | Agent name (URL-encoded)                                               |
| `projectId`        | Yes      | ARM resource ID of the Foundry project (URL-encoded — contains slashes)|
| `top`              | No       | Max insights to return (default 50)                                    |

**Auth:** `az account get-access-token --resource https://ai.azure.com`

**Body:** Must send `{}` (empty JSON object) — POST with no body returns 400.

## Example

```powershell
$token = az account get-access-token --resource https://ai.azure.com --query accessToken -o tsv
$encodedAgent = [uri]::EscapeDataString("my-agent")
$encodedProjectId = [uri]::EscapeDataString("/subscriptions/{sub}/resourceGroups/{rg}/providers/Microsoft.CognitiveServices/accounts/{account}/projects/{project}")

$uri = "https://{region}.api.azureml.ms/notification/v1-beta2/subscriptions/{sub}/resourceGroups/{rg}/providers/microsoft.insights/components/{component}/:insights?startDateTimeUtc=2025-01-01T00:00:00Z&endDateTimeUtc=2025-01-18T00:00:00Z&agent=$encodedAgent&projectId=$encodedProjectId&top=50"

$response = Invoke-RestMethod -Uri $uri -Method POST -Headers @{
    "Authorization" = "Bearer $token"
    "Content-Type" = "application/json"
} -Body "{}"
```

## Response Structure (v1-beta2)

Response is grouped by agent version. Each insight includes `relatedSpans` with `operationId` (App Insights trace ID) for querying full trace content.

```json
{
  "agents": [{
    "agent": "my-agent:1",
    "insights": [{
      "id": "anomaly-token-shift-<hash>",
      "type": "Token",
      "severity": "Critical",
      "message": "Token usage increased by 137%",
      "agentVersion": "1",
      "metadata": { "meanBefore": 2041, "meanAfter": 4831, "confidence": 0.91 },
      "relatedSpans": {
        "totalCount": 13,
        "spans": [
          { "responseId": "resp_...", "operationId": "<trace-id>", "evaluationRunId": null }
        ]
      }
    }],
    "insightCount": 3
  }],
  "totalCount": 3, "criticalCount": 1, "warningCount": 1, "improvementCount": 1
}
```

## Querying Traces from relatedSpans

Use `operationId` from `relatedSpans` to fetch full trace content from App Insights:

```kql
dependencies
| where operation_Id == "<operationId>"
| where customDimensions has "invoke_agent"
| project input = customDimensions["gen_ai.input.messages"],
          output = customDimensions["gen_ai.output.messages"],
          tokens = toint(customDimensions["gen_ai.usage.output_tokens"])
```

This returns the user query and agent response for the specific trace flagged by the insight.

## How Changepoint Detection Works

The API finds **statistical inflection points within the queried time window**. `meanBefore`/`meanAfter` represent averages on either side of the detected shift — not comparisons to a historical baseline.

- 10+ data points give better signal for changepoint detection
- `confidence` close to 1.0 = statistically significant shift

## Next Steps

After receiving insights with `Warning` or `Critical` severity:
1. Use `relatedSpans.operationId` values to query full trace content from App Insights (see KQL above)
2. Present the insights summary to the user with severity, type, evaluator name, and shift magnitude
3. Offer to drill into specific traces for detailed analysis using the [trace analysis skill](../trace.md)

# Foundry Agent Trace Analysis

Analyze production traces for Foundry agents using Application Insights and GenAI OpenTelemetry semantic conventions. This skill provides structured KQL-powered workflows for a selected agent root and environment: searching conversations, diagnosing failures, and identifying latency bottlenecks.

## When to Use This Skill

USE FOR: analyze agent traces, search agent conversations, find failing traces, slow traces, latency analysis, trace search, conversation history, agent errors in production, debug agent responses, App Insights traces, GenAI telemetry, trace correlation, span tree, production trace analysis, evaluation results, evaluation scores, eval run results, find by response ID, get agent trace by conversation ID, agent evaluation scores from App Insights.

> **USE THIS SKILL INSTEAD OF** `azure-monitor` or `azure-applicationinsights` when querying Foundry agent traces, evaluations, or GenAI telemetry. This skill has correct GenAI OTel attribute mappings and tested KQL templates that those general tools lack.

> ⚠️ **DO NOT manually write KQL queries** for GenAI trace analysis **without reading this skill first.** This skill provides tested query templates with correct GenAI OTel attribute mappings, proper span correlation logic, environment-aware scoping, and conversation-level aggregation patterns.

## Quick Reference

| Property | Value |
|----------|-------|
| Data source | Application Insights (App Insights) |
| Query language | KQL (Kusto Query Language) |
| Related skills | `troubleshoot` (hosted-agent logs), `eval-datasets` (trace harvesting) |
| Preferred query tool | `monitor_resource_log_query` (Azure MCP) - use for App Insights KQL queries |
| OTel conventions | [GenAI Spans](https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-spans/), [Agent Spans](https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-agent-spans/) |
| Local metadata | selected `.foundry/agent-metadata*.yaml` overlay/cache file |

## Entry Points

| User Intent | Start At |
|-------------|----------|
| "Search agent conversations" / "Find traces" | [Search Traces](references/search-traces.md) |
| "Tell me about response ID X" / "Look up response ID" | [Search Traces - Search by Response ID](references/search-traces.md#search-by-response-id) |
| "Why is my agent failing?" / "Find errors" | [Analyze Failures](references/analyze-failures.md) |
| "My agent is slow" / "Latency analysis" | [Analyze Latency](references/analyze-latency.md) |
| "Show me this conversation" / "Trace detail" | [Conversation Detail](references/conversation-detail.md) |
| "Find eval results for response ID" / "eval scores from traces" | [Eval Correlation](references/eval-correlation.md) |
| "What KQL do I need?" | [KQL Templates](references/kql-templates.md) |
| "Auto-detect agent issues" / "Get automated insights" / "What's wrong with my agent?" | [Tracing Insights API](references/tracing-insights-api.md) |

## Before Starting — Resolve App Insights Connection

1. Resolve the target agent root, environment, effective deployment context, and selected metadata overlay using [Common Project Context Resolution](../../SKILL.md#agent-common-project-context-resolution).
2. In azd projects, prefer App Insights values from `azd env get-values`; otherwise check `environments.<env>.observability.applicationInsightsConnectionString` or `environments.<env>.observability.applicationInsightsResourceId` in the selected metadata file.
3. If observability settings are missing, use `project_connection_list` to discover App Insights linked to the Foundry project, then persist the chosen resource back to `environments.<env>.observability` only when azd cannot provide it.
4. Confirm the selected App Insights resource and environment with the user before querying.
5. Use **`monitor_resource_log_query`** (Azure MCP tool) to execute KQL queries against the App Insights resource. This is preferred over delegating to the `azure-kusto` skill. Pass the App Insights resource ID and the KQL query directly.

| Metadata field | Purpose | Example |
|----------------|---------|---------|
| `environments.<env>.observability.applicationInsightsConnectionString` | App Insights connection string | `InstrumentationKey=...;IngestionEndpoint=...` |
| `environments.<env>.observability.applicationInsightsResourceId` | ARM resource ID | `/subscriptions/.../Microsoft.Insights/components/...` |

> ⚠️ **Always pass `subscription` explicitly** to Azure MCP tools like `monitor_resource_log_query` - they do not extract it from resource IDs.

## Behavioral Rules

1. **Always display the KQL query.** Before executing any KQL query, display it in a code block. Never run a query silently.
2. **Keep environment visible.** Include the selected environment and agent name in each search summary, and include the derived agent version when the query can recover it from telemetry.
3. **Start broad, then narrow.** Begin with conversation-level summaries, then drill into specific conversations or spans on user request.
4. **Use time ranges.** Always scope queries with a time range (default: last 24 hours). Ask the user for the range if not specified.
5. **Explain GenAI attributes.** When displaying results, translate OTel attribute names to human-readable labels (for example, `gen_ai.operation.name` -> "Operation").
6. **Link to conversation detail.** When showing search or failure results, offer to drill into any specific conversation.
7. **Scope to the selected environment.** App Insights may contain traces from multiple agents or environments. Filter with the selected environment's agent name first, then add an environment tag filter if the telemetry emits one.
8. **Resolve hosted-agent identity from `requests` first.** For hosted agents, prefer `requests`-scoped `gen_ai.agent.name` or `azure.ai.agentserver.agent_name` as the Foundry-facing filter. When `gen_ai.agent.id` is emitted in `<agent-name>:<version>` format, parse it to surface `agentVersion`, but do not treat `dependencies.gen_ai.agent.name` as the top-level hosted-agent name.
9. **Use `operation_Id` to fan out hosted-agent traces.** After isolating the hosted-agent `requests` rows, materialize their `operation_Id` values and join other telemetry tables on `operation_Id`. When conversation IDs are sparse, use `coalesce(gen_ai.conversation.id, azure.ai.agentserver.conversation_id, operation_Id)` so every row still rolls up to a stable conversation key.

# Foundry Agent Troubleshoot

Troubleshoot and debug Foundry agents by collecting hosted-agent session logs, discovering observability connections, and querying Application Insights telemetry.

## Quick Reference

| Property | Value |
|----------|-------|
| Agent types | Prompt (LLM-based), Hosted |
| MCP servers | `azure` |
| Key Foundry MCP tools | `agent_get` |
| Related skills | `trace` (telemetry analysis) |
| Preferred query tool | `monitor_resource_log_query` (Azure MCP) — preferred over `azure-kusto` for App Insights |
| CLI references | `az cognitiveservices account connection`, `az rest`, `curl` |

## When to Use This Skill

- Agent is not responding or returning errors
- Hosted agent version is not becoming active
- Need to view hosted-agent session logs
- Diagnose latency or timeout issues
- Query Application Insights for agent traces and exceptions
- Investigate agent runtime failures

## MCP Tools

| Tool | Description | Parameters |
|------|-------------|------------|
| `agent_get` | Get agent details to determine type and inspect agent/version status | `projectEndpoint` (required), `agentName` (optional) |

## Workflow

### Step 1: Collect Agent Information

Use the project endpoint and agent name from the project context (see [Common Project Context Resolution](../../SKILL.md#agent-common-project-context-resolution)). Ask the user only for values not already resolved:
- **Project endpoint** — AI Foundry project endpoint URL
- **Agent name** — Name of the agent to troubleshoot

### Step 2: Determine Agent Type

Use `agent_get` with `projectEndpoint` and `agentName` to retrieve the agent definition. Check the `kind` field:
- `"hosted"` → Proceed to Step 3
- `"prompt"` → Skip to Step 4 (Discover Observability Connections)

### Step 3: Retrieve Logs (Hosted Agents Only)

Hosted-agent logs are scoped to individual **sessions** (sandbox instances).

> ℹ️ **`invocations_ws` agents:** the `sessionId` used by these REST endpoints is the **client-supplied `agent_session_id`** that the WebSocket client put on the upgrade URL — not a value issued by `session_create`. If the user has the WS client logs, pull the `agent_session_id` from there and pass it as `sessionId` below. See the [invocations-ws skill](../invocations-ws/invocations-ws.md) for the WS URL contract.

1. **Check agent version status** — Use `agent_get` to verify the agent version status is `active`. If it is not active, the agent may still be provisioning or may have failed to become active.

2. **List sessions** — Hosted-agent logs require a `sessionId`. If the user does not have one, list available sessions:
   ```bash
   az rest --method GET \
     --url "<projectEndpoint>/agents/<agentName>/sessions?api-version=2025-11-15-preview" \
     --headers "Foundry-Features=HostedAgents=V1Preview" \
     --resource "https://ai.azure.com"
   ```

3. **Retrieve session logs** — The log stream endpoint uses Server-Sent Events (SSE). Use `curl` with a timeout:
   ```bash
   TOKEN=$(az account get-access-token --resource "https://ai.azure.com" --query accessToken -o tsv)
   curl -s --max-time 15 \
     -H "Authorization: Bearer $TOKEN" \
     -H "Accept: text/event-stream" \
     -H "Foundry-Features: HostedAgents=V1Preview" \
     "<projectEndpoint>/agents/<agentName>/sessions/<sessionId>:logstream?api-version=2025-11-15-preview"
   ```

   > ⚠️ **404 is expected** if the session sandbox has not been created yet. Advise the user to send a message to the agent first to trigger sandbox creation, then retry.

4. **Interpret the logs** — Each SSE frame is `event: log\ndata: {...}\n\n`:
   - **Preamble** (first event): JSON with `session_state`, `session_id`, `agent`, `version`, `last_accessed`
   - **Log lines** (subsequent events): JSON with `stream` (`stdout`/`stderr`/`status`), `message`, and `timestamp`
   - **Error events**: `event: error` frames indicate server-side errors within the session sandbox

   Present the logs to the user and highlight any errors or warnings found.

### Step 4: Discover Observability Connections

List the project connections to find Application Insights or Azure Monitor resources using the Azure CLI command documented at:
[az cognitiveservices account connection](https://learn.microsoft.com/en-us/cli/azure/cognitiveservices/account/connection?view=azure-cli-latest)

Refer to the documentation above for the exact command syntax and parameters. Look for connections of type `ApplicationInsights` or `AzureMonitor` in the output.

If no observability connection is found, inform the user and suggest setting up Application Insights for the project. Ask if they want to proceed without telemetry data.

### Step 5: Query Application Insights Telemetry

Use **`monitor_resource_log_query`** (Azure MCP tool) to run KQL queries against the Application Insights resource discovered in Step 4. This is preferred over delegating to the `azure-kusto` skill. Pass the App Insights resource ID and the KQL query directly.

> ⚠️ **Always pass `subscription` explicitly** to Azure MCP tools like `monitor_resource_log_query` — they don't extract it from resource IDs.

Use `* contains "<response_id>"` or `* contains "<agent_name>"` filters to narrow down results to the specific agent instance.

### Step 6: Summarize Findings

Present a summary to the user including:
- **Agent type and status** — hosted or prompt; hosted agent version status when relevant
- **Log errors** — key errors from hosted-agent session logs
- **Telemetry insights** — exceptions, failed requests, latency trends
- **Recommended actions** — specific steps to resolve identified issues

## Error Handling

| Error | Cause | Resolution |
|-------|-------|------------|
| Agent not found | Invalid agent name or project endpoint | Use `agent_get` to list available agents and verify name |
| Hosted agent not active | Hosted agent is still provisioning or failed | Check that the ACR image was pushed correctly and agent identity permissions are assigned; wait and re-check status |
| Session logs 404 | Session sandbox has not been created yet | The sandbox is created on first invocation — send a message to the agent to trigger sandbox creation, then retry |
| SSE error event | Server-side error within the session sandbox | Check the error event `data` field for details |
| No session ID | User does not know which session to troubleshoot | List sessions via REST API (see Step 3) |
| No observability connection | Application Insights not configured for the project | Suggest configuring Application Insights for the Foundry project |
| Kusto query failed | Invalid cluster/database or insufficient permissions | Verify Application Insights resource details and reader permissions |
| No telemetry data | Agent not instrumented or too recent | Check if Application Insights SDK is configured; data may take a few minutes to appear |

## Additional Resources

- [Foundry Hosted Agents](https://learn.microsoft.com/azure/ai-foundry/agents/concepts/hosted-agents?view=foundry)
- [Account Connection CLI Reference](https://learn.microsoft.com/en-us/cli/azure/cognitiveservices/account/connection?view=azure-cli-latest)
- [KQL Quick Reference](https://learn.microsoft.com/azure/data-explorer/kusto/query/kql-quick-reference)
- [Foundry Samples](https://github.com/microsoft-foundry/foundry-samples)

---
name: deploy-model
description: "Unified Azure OpenAI model deployment skill with intelligent intent-based routing. Handles quick preset deployments, fully customized deployments (version/SKU/capacity/RAI policy), and capacity discovery across regions and projects. USE FOR: deploy model, deploy gpt, create deployment, model deployment, deploy openai model, set up model, provision model, find capacity, check model availability, where can I deploy, best region for model, capacity analysis. DO NOT USE FOR: listing existing deployments (use foundry_models_deployments_list MCP tool), deleting deployments, agent creation (use agent/create), project creation (use project/create)."
license: MIT
metadata:
  author: Microsoft
  version: "1.0.0"
---

# Deploy Model

> **Scope — read this first.** This skill creates model deployments **out-of-band** via Azure CLI / MCP / portal. For azd-managed Foundry projects (those scaffolded from `azd-ai-starter-basic` or via `azd ai agent init`), declare deployments in `azure.yaml services.ai-project.deployments[]` instead — `azd ai agent init` writes the entry from the sample manifest and `azd provision` creates the deployment through Bicep. See [foundry-agent/create/create-hosted.md](../../foundry-agent/create/create-hosted.md) for the Golden Path. Use this skill only for: (a) Foundry projects not managed by an azd project, (b) ad-hoc deployments outside the azd lifecycle.

Unified entry point for all Azure OpenAI model deployment workflows. Analyzes user intent and routes to the appropriate deployment mode.

## Quick Reference

| Mode | When to Use | Sub-Skill |
|------|-------------|-----------|
| **Preset** | Quick deployment, no customization needed | [preset/SKILL.md](preset/SKILL.md) |
| **Customize** | Full control: version, SKU, capacity, RAI policy | [customize/SKILL.md](customize/SKILL.md) |
| **Capacity Discovery** | Find where you can deploy with specific capacity | [capacity/SKILL.md](capacity/SKILL.md) |

## Intent Detection

Analyze the user's prompt and route to the correct mode:

```
User Prompt
    │
    ├─ Simple deployment (no modifiers)
    │  "deploy gpt-4o", "set up a model"
    │  └─> PRESET mode
    │
    ├─ Customization keywords present
    │  "custom settings", "choose version", "select SKU",
    │  "set capacity to X", "configure content filter",
    │  "PTU deployment", "with specific quota"
    │  └─> CUSTOMIZE mode
    │
    ├─ Capacity/availability query
    │  "find where I can deploy", "check capacity",
    │  "which region has X capacity", "best region for 10K TPM",
    │  "where is this model available"
    │  └─> CAPACITY DISCOVERY mode
    │
    └─ Ambiguous (has capacity target + deploy intent)
       "deploy gpt-4o with 10K capacity to best region"
       └─> CAPACITY DISCOVERY first → then PRESET or CUSTOMIZE
```

### Routing Rules

| Signal in Prompt | Route To | Reason |
|------------------|----------|--------|
| Just model name, no options | **Preset** | User wants quick deployment |
| "custom", "configure", "choose", "select" | **Customize** | User wants control |
| "find", "check", "where", "which region", "available" | **Capacity** | User wants discovery |
| Specific capacity number + "best region" | **Capacity → Preset** | Discover then deploy quickly |
| Specific capacity number + "custom" keywords | **Capacity → Customize** | Discover then deploy with options |
| "PTU", "provisioned throughput" | **Customize** | PTU requires SKU selection |
| "optimal region", "best region" (no capacity target) | **Preset** | Region optimization is preset's specialty |

### Multi-Mode Chaining

Some prompts require two modes in sequence:

**Pattern: Capacity → Deploy**
When a user specifies a capacity requirement AND wants deployment:
1. Run **Capacity Discovery** to find regions/projects with sufficient quota
2. Present findings to user
3. Ask: "Would you like to deploy with **quick defaults** or **customize settings**?"
4. Route to **Preset** or **Customize** based on answer

> 💡 **Tip:** If unsure which mode the user wants, default to **Preset** (quick deployment). Users who want customization will typically use explicit keywords like "custom", "configure", or "with specific settings".

## Project Selection (All Modes)

Before any deployment, resolve which project to deploy to. This applies to **all** modes (preset, customize, and after capacity discovery).

### Resolution Order

1. **Check `PROJECT_RESOURCE_ID` env var** — if set, use it as the default
2. **Check user prompt** — if user named a specific project or region, use that
3. **If neither** — query the user's projects and suggest the current one

### Confirmation Step (Required)

**Always confirm the target before deploying.** Show the user what will be used and give them a chance to change it:

```
Deploying to:
  Project:  <project-name>
  Region:   <region>
  Resource: <resource-group>

Is this correct? Or choose a different project:
  1. ✅ Yes, deploy here (default)
  2. 📋 Show me other projects in this region
  3. 🌍 Choose a different region
```

If user picks option 2, show top 5 projects in that region:

```
Projects in <region>:
  1. project-alpha (rg-alpha)
  2. project-beta (rg-beta)
  3. project-gamma (rg-gamma)
  ...
```

> ⚠️ **Never deploy without showing the user which project will be used.** This prevents accidental deployments to the wrong resource.

## Pre-Deployment Validation (All Modes)

Before presenting any deployment options (SKU, capacity), always validate both of these:

1. **Model supports the SKU** — query the model catalog to confirm the selected model+version supports the target SKU:
   ```bash
   az cognitiveservices model list --location <region> --subscription <sub-id> -o json
   ```
   Filter for the model, extract `.model.skus[].name` to get supported SKUs.

2. **Subscription has available quota** — check that the user's subscription has unallocated quota for the SKU+model combination:
   ```bash
   az cognitiveservices usage list --location <region> --subscription <sub-id> -o json
   ```
   Match by usage name pattern `OpenAI.<SKU>.<model-name>` (e.g., `OpenAI.GlobalStandard.gpt-4o`). Compute `available = limit - currentValue`.

> ⚠️ **Warning:** Only present options that pass both checks. Do NOT show hardcoded SKU lists — always query dynamically. SKUs with 0 available quota should be shown as ❌ informational items, not selectable options.

> 💡 **Quota management:** For quota increase requests, usage monitoring, and troubleshooting quota errors, defer to the [quota skill](../../quota/quota.md) instead of duplicating that guidance inline.

## Prerequisites

All deployment modes require:
- Azure CLI installed and authenticated (`az login`)
- Active Azure subscription with deployment permissions
- Azure AI Foundry project resource ID (or agent will help discover it via `PROJECT_RESOURCE_ID` env var)

## Sub-Skills

- **[preset/SKILL.md](preset/SKILL.md)** — Quick deployment to optimal region with sensible defaults
- **[customize/SKILL.md](customize/SKILL.md)** — Interactive guided flow with full configuration control
- **[capacity/SKILL.md](capacity/SKILL.md)** — Discover available capacity across regions and projects

# Deploy Model — Test Prompts

Test prompts for the unified `deploy-model` skill with router, preset, customize, and capacity sub-skills.

## Preset Mode (Quick Deploy)

| # | Prompt | Expected |
|---|--------|----------|
| 1 | Deploy gpt-4o | Preset — confirm project, deploy with defaults |
| 2 | Set up o3-mini for me | Preset — pick latest version automatically |
| 3 | I need a text-embedding-ada-002 deployment | Preset — non-chat model |
| 4 | Deploy gpt-4o to the best region | Preset — region scan, no capacity target |

## Customize Mode (Guided Flow)

| # | Prompt | Expected |
|---|--------|----------|
| 5 | Deploy gpt-4o with custom settings | Customize — walk through version → SKU → capacity → RAI |
| 6 | I want to choose the version and SKU for my o3-mini deployment | Customize — explicit keywords |
| 7 | Set up a PTU deployment for gpt-4o | Customize — PTU requires SKU selection |
| 8 | Deploy gpt-4o with a specific content filter | Customize — RAI policy flow |

## Capacity Discovery

| # | Prompt | Expected |
|---|--------|----------|
| 9 | Where can I deploy gpt-4o? | Capacity — show regions, no deploy |
| 10 | Which regions have o3-mini available? | Capacity — run script, show table |
| 11 | Check if I have enough quota for gpt-4o with 500K TPM | Capacity — high target, some regions may not qualify |

## Chained (Capacity → Deploy)

| # | Prompt | Expected |
|---|--------|----------|
| 12 | Find me the best region and project to deploy gpt-4o with 10K capacity | Capacity → Preset |
| 13 | Deploy o3-mini with 200K TPM to whatever region has it | Capacity → Preset |
| 14 | I want to deploy gpt-4o with 50K capacity and choose my own settings | Capacity → Customize |

## Negative / Edge Cases

| # | Prompt | Expected |
|---|--------|----------|
| 15 | Deploy unicorn-model-9000 | Fail gracefully — model doesn't exist |
| 16 | Deploy gpt-4o with 999999K TPM | Capacity shows no region qualifies |
| 17 | Deploy gpt-4o (with az login expired) | Auth error caught early |
| 18 | Delete my gpt-4o deployment | Should NOT trigger deploy-model |
| 19 | List my current deployments | Should NOT trigger deploy-model |
| 20 | Deploy gpt-4o to mars-region-1 | Fail gracefully — invalid region |

## Project Selection

| # | Prompt | Expected |
|---|--------|----------|
| 21 | Deploy gpt-4o (with PROJECT_RESOURCE_ID set) | Show current project, confirm before deploying |
| 22 | Deploy gpt-4o (no PROJECT_RESOURCE_ID) | Ask user to pick a project |
| 23 | Deploy gpt-4o to project my-special-project | Use named project directly |

## Ambiguous / Routing Stress

| # | Prompt | Expected |
|---|--------|----------|
| 24 | Help me with model deployment | Preset (default) — vague, no keywords |
| 25 | I need gpt-4o deployed fast with good capacity | Preset — "fast" + vague capacity |
| 26 | Can you configure a deployment? | Customize — "configure" keyword, should ask which model |
| 27 | What's the best way to deploy gpt-4o with 100K? | Capacity → Preset |

## Automated Test Results (2026-02-09)

All 18 tests passed. Deployments created during testing were cleaned up.

| Category | Tests | Result |
|----------|-------|--------|
| Preset | 3/3 | ✅ |
| Customize | 2/2 | ✅ |
| Capacity | 3/3 | ✅ |
| Chained | 1/1 | ✅ |
| Negative | 5/5 | ✅ |
| Ambiguous | 4/4 | ✅ |

---
name: capacity
description: "Discovers available Azure OpenAI model capacity across regions and projects. Analyzes quota limits, compares availability, and recommends optimal deployment locations based on capacity requirements. USE FOR: find capacity, check quota, where can I deploy, capacity discovery, best region for capacity, multi-project capacity search, quota analysis, model availability, region comparison, check TPM availability. DO NOT USE FOR: actual deployment (hand off to preset or customize after discovery), quota increase requests (direct user to Azure Portal), listing existing deployments."
license: MIT
metadata:
  author: Microsoft
  version: "1.0.0"
---

# Capacity Discovery

Finds available Azure OpenAI model capacity across all accessible regions and projects. Recommends the best deployment location based on capacity requirements.

## Quick Reference

| Property | Description |
|----------|-------------|
| **Purpose** | Find where you can deploy a model with sufficient capacity |
| **Scope** | All regions and projects the user has access to |
| **Output** | Ranked table of regions/projects with available capacity |
| **Action** | Read-only analysis — does NOT deploy. Hands off to preset or customize |
| **Authentication** | Azure CLI (`az login`) |

## When to Use This Skill

- ✅ User asks "where can I deploy gpt-4o?"
- ✅ User specifies a capacity target: "find a region with 10K TPM for gpt-4o"
- ✅ User wants to compare availability: "which regions have gpt-4o available?"
- ✅ User got a quota error and needs to find an alternative location
- ✅ User asks "best region and project for deploying model X"

**After discovery → hand off to [preset](../preset/SKILL.md) or [customize](../customize/SKILL.md) for actual deployment.**

## Scripts

Pre-built scripts handle the complex REST API calls and data processing. Use these instead of constructing commands manually.

| Script | Purpose | Usage |
|--------|---------|-------|
| `scripts/discover_and_rank.ps1` | Full discovery: capacity + projects + ranking | Primary script for capacity discovery |
| `scripts/discover_and_rank.sh` | Same as above (bash) | Primary script for capacity discovery |
| `scripts/query_capacity.ps1` | Raw capacity query (no project matching) | Quick capacity check or version listing |
| `scripts/query_capacity.sh` | Same as above (bash) | Quick capacity check or version listing |

## Workflow

### Phase 1: Validate Prerequisites

```bash
az account show --query "{Subscription:name, SubscriptionId:id}" --output table
```

### Phase 2: Identify Model and Version

Extract model name from user prompt. If version is unknown, query available versions:

```powershell
.\scripts\query_capacity.ps1 -ModelName <model-name>
```
```bash
./scripts/query_capacity.sh <model-name>
```

This lists available versions. Use the latest version unless user specifies otherwise.

### Phase 3: Run Discovery

Run the full discovery script with model name, version, and minimum capacity target:

```powershell
.\scripts\discover_and_rank.ps1 -ModelName <model-name> -ModelVersion <version> -MinCapacity <target>
```
```bash
./scripts/discover_and_rank.sh <model-name> <version> <min-capacity>
```

> 💡 The script automatically queries capacity across ALL regions, cross-references with the user's existing projects, and outputs a ranked table sorted by: meets target → project count → available capacity.

### Phase 3.5: Validate Subscription Quota

After discovery identifies candidate regions, validate that the user's subscription actually has available quota in each region. Model capacity (from Phase 3) shows what the platform can support, but subscription quota limits what this specific user can deploy.

```powershell
# For each candidate region from discovery results:
$usageData = az cognitiveservices usage list --location <region> --subscription $SUBSCRIPTION_ID -o json 2>$null | ConvertFrom-Json

# Check quota for each SKU the model supports
# Quota names follow pattern: OpenAI.<SKU>.<model-name>
$usageEntry = $usageData | Where-Object { $_.name.value -eq "OpenAI.<SKU>.<model-name>" }

if ($usageEntry) {
  $quotaAvailable = $usageEntry.limit - $usageEntry.currentValue
} else {
  $quotaAvailable = 0  # No quota allocated
}
```
```bash
# For each candidate region from discovery results:
usage_json=$(az cognitiveservices usage list --location <region> --subscription "$SUBSCRIPTION_ID" -o json 2>/dev/null)

# Extract quota for specific SKU+model
quota_available=$(echo "$usage_json" | jq -r --arg name "OpenAI.<SKU>.<model-name>" \
  '.[] | select(.name.value == $name) | .limit - .currentValue')
```

**Annotate discovery results:**

Add a "Quota Available" column to the ranked output from Phase 3:

| Region | Available Capacity | Meets Target | Projects | Quota Available |
|--------|-------------------|--------------|----------|-----------------|
| eastus2 | 120K TPM | ✅ | 3 | ✅ 80K |
| westus3 | 90K TPM | ✅ | 1 | ❌ 0 (at limit) |
| swedencentral | 100K TPM | ✅ | 0 | ✅ 100K |

Regions/SKUs where `quotaAvailable = 0` should be marked with ❌ in the results. If no region has available quota, hand off to the [quota skill](../../../quota/quota.md) for increase requests and troubleshooting.

### Phase 4: Present Results and Hand Off

After the script outputs the ranked table (now annotated with quota info), present it to the user and ask:

1. 🚀 **Quick deploy** to top recommendation with defaults → route to [preset](../preset/SKILL.md)
2. ⚙️ **Custom deploy** with version/SKU/capacity/RAI selection → route to [customize](../customize/SKILL.md)
3. 📊 **Check another model** or capacity target → re-run Phase 2
4. ❌ Cancel

### Phase 5: Confirm Project Before Deploying

Before handing off to preset or customize, **always confirm the target project** with the user. See the [Project Selection](../SKILL.md#project-selection-all-modes) rules in the parent router.

If the discovery table shows a sample project for the chosen region, suggest it as the default. Otherwise, query projects in that region and let the user pick.

## Error Handling

| Error | Cause | Resolution |
|-------|-------|------------|
| "No capacity found" | Model not available or all at quota | Hand off to [quota skill](../../../quota/quota.md) for increase requests and troubleshooting |
| Script auth error | `az login` expired | Re-run `az login` |
| Empty version list | Model not in region catalog | Try a different region: `./scripts/query_capacity.sh <model> "" eastus` |
| "No projects found" | No AI Services resources | Guide to `project/create` skill or Azure Portal |

## Related Skills

- **[preset](../preset/SKILL.md)** — Quick deployment after capacity discovery
- **[customize](../customize/SKILL.md)** — Custom deployment after capacity discovery
- **[quota](../../../quota/quota.md)** — For quota viewing, increase requests, and troubleshooting quota errors, defer to this skill instead of duplicating guidance

<#
.SYNOPSIS
    Discovers available capacity for an Azure OpenAI model across all regions,
    cross-references with existing projects and subscription quota, and outputs a ranked table.
.PARAMETER ModelName
    The model name (e.g., "gpt-4o", "o3-mini")
.PARAMETER ModelVersion
    The model version (e.g., "2025-01-31")
.PARAMETER MinCapacity
    Minimum required capacity in K TPM units (default: 0, shows all)
.EXAMPLE
    .\discover_and_rank.ps1 -ModelName o3-mini -ModelVersion 2025-01-31 -MinCapacity 200
#>
param(
    [Parameter(Mandatory)][string]$ModelName,
    [Parameter(Mandatory)][string]$ModelVersion,
    [int]$MinCapacity = 0
)

$ErrorActionPreference = "Stop"

$subId = az account show --query id -o tsv

# Query model capacity across all regions
$capRaw = az rest --method GET `
    --url "https://management.azure.com/subscriptions/$subId/providers/Microsoft.CognitiveServices/modelCapacities" `
    --url-parameters api-version=2024-10-01 modelFormat=OpenAI modelName=$ModelName modelVersion=$ModelVersion `
    2>$null | Out-String | ConvertFrom-Json

# Query all AI Foundry projects (AIProject kind)
$projRaw = az rest --method GET `
    --url "https://management.azure.com/subscriptions/$subId/providers/Microsoft.CognitiveServices/accounts" `
    --url-parameters api-version=2024-10-01 `
    --query "value[?kind=='AIProject'].{Name:name, Location:location}" `
    2>$null | Out-String | ConvertFrom-Json

# Build capacity map (GlobalStandard only, pick max per region)
$capMap = @{}
foreach ($item in $capRaw.value) {
    $sku = $item.properties.skuName
    $avail = [int]$item.properties.availableCapacity
    $region = $item.location
    if ($sku -eq "GlobalStandard" -and $avail -gt 0) {
        if (-not $capMap[$region] -or $avail -gt $capMap[$region]) {
            $capMap[$region] = $avail
        }
    }
}

# Build project map
$projMap = @{}
$projSample = @{}
foreach ($p in $projRaw) {
    $loc = $p.Location
    if (-not $projMap[$loc]) { $projMap[$loc] = 0 }
    $projMap[$loc]++
    if (-not $projSample[$loc]) { $projSample[$loc] = $p.Name }
}

# Check subscription quota per region
$quotaMap = @{}
$checkedRegions = @{}
foreach ($region in $capMap.Keys) {
    if ($checkedRegions[$region]) { continue }
    $checkedRegions[$region] = $true
    try {
        $usageData = az cognitiveservices usage list --location $region --subscription $subId -o json 2>$null | Out-String | ConvertFrom-Json
        $usageEntry = $usageData | Where-Object { $_.name.value -eq "OpenAI.GlobalStandard.$ModelName" }
        if ($usageEntry) {
            $quotaMap[$region] = [int]$usageEntry.limit - [int]$usageEntry.currentValue
        } else {
            $quotaMap[$region] = 0
        }
    } catch {
        $quotaMap[$region] = -1  # Unable to check
    }
}

# Combine and rank
$results = foreach ($region in $capMap.Keys) {
    $avail = $capMap[$region]
    $meets = $avail -ge $MinCapacity
    $quota = if ($quotaMap[$region]) { $quotaMap[$region] } else { 0 }
    $quotaDisplay = if ($quota -eq -1) { "?" } elseif ($quota -gt 0) { "${quota}K" } else { "0" }
    $quotaOk = $quota -gt 0 -or $quota -eq -1
    [PSCustomObject]@{
        Region         = $region
        AvailableTPM   = "${avail}K"
        AvailableRaw   = $avail
        MeetsTarget    = if ($meets) { "YES" } else { "no" }
        Projects       = if ($projMap[$region]) { $projMap[$region] } else { 0 }
        SampleProject  = if ($projSample[$region]) { $projSample[$region] } else { "(none)" }
        QuotaAvailable = $quotaDisplay
        QuotaOk        = $quotaOk
    }
}

$results = $results | Sort-Object @{Expression={$_.MeetsTarget -eq "YES"}; Descending=$true},
                                   @{Expression={$_.QuotaOk}; Descending=$true},
                                   @{Expression={$_.Projects}; Descending=$true},
                                   @{Expression={$_.AvailableRaw}; Descending=$true}

# Output summary
$total = ($results | Measure-Object).Count
$matching = ($results | Where-Object { $_.MeetsTarget -eq "YES" } | Measure-Object).Count
$withQuota = ($results | Where-Object { $_.MeetsTarget -eq "YES" -and $_.QuotaOk } | Measure-Object).Count
$withProjects = ($results | Where-Object { $_.MeetsTarget -eq "YES" -and $_.Projects -gt 0 } | Measure-Object).Count

Write-Host "Model: $ModelName v$ModelVersion | SKU: GlobalStandard | Min Capacity: ${MinCapacity}K TPM"
Write-Host "Regions with capacity: $total | Meets target: $matching | With quota: $withQuota | With projects: $withProjects"
Write-Host ""

$results | Select-Object Region, AvailableTPM, MeetsTarget, QuotaAvailable, Projects, SampleProject | Format-Table -AutoSize

#!/bin/bash
# discover_and_rank.sh
# Discovers available capacity for an Azure OpenAI model across all regions,
# cross-references with existing projects and subscription quota, and outputs a ranked table.
#
# Usage: ./discover_and_rank.sh <model-name> <model-version> [min-capacity]
# Example: ./discover_and_rank.sh o3-mini 2025-01-31 200
#
# Output: Ranked table of regions with capacity, quota, project counts, and match status

set -euo pipefail

MODEL_NAME="${1:?Usage: $0 <model-name> <model-version> [min-capacity]}"
MODEL_VERSION="${2:?Usage: $0 <model-name> <model-version> [min-capacity]}"
MIN_CAPACITY="${3:-0}"

SUB_ID=$(az account show --query id -o tsv)

# Query model capacity across all regions (GlobalStandard SKU)
CAPACITY_JSON=$(az rest --method GET \
  --url "https://management.azure.com/subscriptions/${SUB_ID}/providers/Microsoft.CognitiveServices/modelCapacities" \
  --url-parameters api-version=2024-10-01 modelFormat=OpenAI modelName="$MODEL_NAME" modelVersion="$MODEL_VERSION" \
  2>/dev/null)

# Query all AI Services projects
PROJECTS_JSON=$(az rest --method GET \
  --url "https://management.azure.com/subscriptions/${SUB_ID}/providers/Microsoft.CognitiveServices/accounts" \
  --url-parameters api-version=2024-10-01 \
  --query "value[?kind=='AIServices'].{name:name, location:location}" \
  2>/dev/null)

# Get unique regions from capacity results for quota checking
REGIONS=$(echo "$CAPACITY_JSON" | jq -r '.value[] | select(.properties.skuName=="GlobalStandard" and .properties.availableCapacity > 0) | .location' | sort -u)

# Build quota map: check subscription quota per region
declare -A QUOTA_MAP
for region in $REGIONS; do
  usage_json=$(az cognitiveservices usage list --location "$region" --subscription "$SUB_ID" -o json 2>/dev/null || echo "[]")
  quota_avail=$(echo "$usage_json" | jq -r --arg name "OpenAI.GlobalStandard.$MODEL_NAME" \
    '[.[] | select(.name.value == $name)] | if length > 0 then .[0].limit - .[0].currentValue else 0 end')
  QUOTA_MAP[$region]="${quota_avail:-0}"
done

# Export quota map as JSON for Python
QUOTA_JSON="{"
first=true
for region in "${!QUOTA_MAP[@]}"; do
  if [ "$first" = true ]; then first=false; else QUOTA_JSON+=","; fi
  QUOTA_JSON+="\"$region\":${QUOTA_MAP[$region]}"
done
QUOTA_JSON+="}"

# Combine, rank, and output using inline Python (available on all Azure CLI installs)
python3 -c "
import json, sys

capacity = json.loads('''${CAPACITY_JSON}''')
projects = json.loads('''${PROJECTS_JSON}''')
quota = json.loads('''${QUOTA_JSON}''')
min_cap = int('${MIN_CAPACITY}')

# Build capacity map (GlobalStandard only)
cap_map = {}
for item in capacity.get('value', []):
    props = item.get('properties', {})
    if props.get('skuName') == 'GlobalStandard' and props.get('availableCapacity', 0) > 0:
        region = item.get('location', '')
        cap_map[region] = max(cap_map.get(region, 0), props['availableCapacity'])

# Build project count map
proj_map = {}
proj_sample = {}
for p in (projects if isinstance(projects, list) else []):
    loc = p.get('location', '')
    proj_map[loc] = proj_map.get(loc, 0) + 1
    if loc not in proj_sample:
        proj_sample[loc] = p.get('name', '')

# Combine and rank
results = []
for region, cap in cap_map.items():
    meets = cap >= min_cap
    q = quota.get(region, 0)
    quota_ok = q > 0
    results.append({
        'region': region,
        'available': cap,
        'meets': meets,
        'projects': proj_map.get(region, 0),
        'sample': proj_sample.get(region, '(none)'),
        'quota': q,
        'quota_ok': quota_ok
    })

# Sort: meets target first, then quota available, then by project count, then by capacity
results.sort(key=lambda x: (-x['meets'], -x['quota_ok'], -x['projects'], -x['available']))

# Output
total = len(results)
matching = sum(1 for r in results if r['meets'])
with_quota = sum(1 for r in results if r['meets'] and r['quota_ok'])
with_projects = sum(1 for r in results if r['meets'] and r['projects'] > 0)

print(f'Model: {\"${MODEL_NAME}\"} v{\"${MODEL_VERSION}\"} | SKU: GlobalStandard | Min Capacity: {min_cap}K TPM')
print(f'Regions with capacity: {total} | Meets target: {matching} | With quota: {with_quota} | With projects: {with_projects}')
print()
print(f'{\"Region\":<22} {\"Available\":<12} {\"Meets Target\":<14} {\"Quota\":<12} {\"Projects\":<10} {\"Sample Project\"}')
print('-' * 100)
for r in results:
    mark = 'YES' if r['meets'] else 'no'
    q_display = f'{r[\"quota\"]}K' if r['quota'] > 0 else '0 (none)'
    print(f'{r[\"region\"]:<22} {r[\"available\"]}K{\"\":.<10} {mark:<14} {q_display:<12} {r[\"projects\"]:<10} {r[\"sample\"]}')
"

<#
.SYNOPSIS
    Queries available capacity for an Azure OpenAI model and validates if a target is achievable.
.PARAMETER ModelName
    The model name (e.g., "gpt-4o", "o3-mini")
.PARAMETER ModelVersion
    The model version (e.g., "2025-01-31"). If omitted, lists available versions.
.PARAMETER Region
    Optional. Check capacity in a specific region only.
.PARAMETER SKU
    SKU to check (default: GlobalStandard)
.EXAMPLE
    .\query_capacity.ps1 -ModelName o3-mini
    .\query_capacity.ps1 -ModelName o3-mini -ModelVersion 2025-01-31 -Region eastus2
#>
param(
    [Parameter(Mandatory)][string]$ModelName,
    [string]$ModelVersion,
    [string]$Region,
    [string]$SKU = "GlobalStandard"
)

$ErrorActionPreference = "Stop"

$subId = az account show --query id -o tsv

# If no version provided, list available versions first
if (-not $ModelVersion) {
    Write-Host "Available versions for $ModelName`:"
    $loc = if ($Region) { $Region } else { "eastus" }
    az cognitiveservices model list --location $loc `
        --query "[?model.name=='$ModelName'].{Version:model.version, Format:model.format}" `
        --output table 2>$null
    return
}

# Build URL parameters
$urlParams = @("api-version=2024-10-01", "modelFormat=OpenAI", "modelName=$ModelName", "modelVersion=$ModelVersion")

if ($Region) {
    $url = "https://management.azure.com/subscriptions/$subId/providers/Microsoft.CognitiveServices/locations/$Region/modelCapacities"
} else {
    $url = "https://management.azure.com/subscriptions/$subId/providers/Microsoft.CognitiveServices/modelCapacities"
}

$raw = az rest --method GET --url $url --url-parameters @urlParams 2>$null | Out-String | ConvertFrom-Json

# Filter by SKU
$filtered = $raw.value | Where-Object { $_.properties.skuName -eq $SKU -and $_.properties.availableCapacity -gt 0 }

if (-not $filtered) {
    Write-Host "No capacity found for $ModelName v$ModelVersion ($SKU)" -ForegroundColor Red
    Write-Host "Try a different SKU or version."
    return
}

Write-Host "Capacity: $ModelName v$ModelVersion ($SKU)"
Write-Host ""
$filtered | ForEach-Object {
    # Check subscription quota for this region
    $quotaDisplay = "?"
    try {
        $usageData = az cognitiveservices usage list --location $_.location --subscription $subId -o json 2>$null | Out-String | ConvertFrom-Json
        $usageEntry = $usageData | Where-Object { $_.name.value -eq "OpenAI.$SKU.$ModelName" }
        if ($usageEntry) {
            $quotaAvail = [int]$usageEntry.limit - [int]$usageEntry.currentValue
            $quotaDisplay = if ($quotaAvail -gt 0) { "${quotaAvail}K" } else { "0 (at limit)" }
        } else {
            $quotaDisplay = "0 (none)"
        }
    } catch {
        $quotaDisplay = "?"
    }
    [PSCustomObject]@{
        Region    = $_.location
        SKU       = $_.properties.skuName
        Available = "$($_.properties.availableCapacity)K TPM"
        Quota     = $quotaDisplay
    }
} | Sort-Object { [int]($_.Available -replace '[^\d]','') } -Descending | Format-Table -AutoSize

#!/bin/bash
# query_capacity.sh
# Queries available capacity for an Azure OpenAI model.
#
# Usage:
#   ./query_capacity.sh <model-name> [model-version] [region] [sku]
# Examples:
#   ./query_capacity.sh o3-mini                          # List versions
#   ./query_capacity.sh o3-mini 2025-01-31               # All regions
#   ./query_capacity.sh o3-mini 2025-01-31 eastus2       # Specific region
#   ./query_capacity.sh o3-mini 2025-01-31 "" Standard   # Different SKU

set -euo pipefail

MODEL_NAME="${1:?Usage: $0 <model-name> [model-version] [region] [sku]}"
MODEL_VERSION="${2:-}"
REGION="${3:-}"
SKU="${4:-GlobalStandard}"

SUB_ID=$(az account show --query id -o tsv)

# If no version, list available versions
if [ -z "$MODEL_VERSION" ]; then
    LOC="${REGION:-eastus}"
    echo "Available versions for $MODEL_NAME:"
    az cognitiveservices model list --location "$LOC" \
        --query "[?model.name=='$MODEL_NAME'].{Version:model.version, Format:model.format}" \
        --output table 2>/dev/null
    exit 0
fi

# Build URL
if [ -n "$REGION" ]; then
    URL="https://management.azure.com/subscriptions/${SUB_ID}/providers/Microsoft.CognitiveServices/locations/${REGION}/modelCapacities"
else
    URL="https://management.azure.com/subscriptions/${SUB_ID}/providers/Microsoft.CognitiveServices/modelCapacities"
fi

# Query capacity
CAPACITY_RESULT=$(az rest --method GET --url "$URL" \
    --url-parameters api-version=2024-10-01 modelFormat=OpenAI modelName="$MODEL_NAME" modelVersion="$MODEL_VERSION" \
    2>/dev/null)

# Get regions with capacity
REGIONS_WITH_CAP=$(echo "$CAPACITY_RESULT" | jq -r ".value[] | select(.properties.skuName==\"$SKU\" and .properties.availableCapacity > 0) | .location" 2>/dev/null | sort -u)

if [ -z "$REGIONS_WITH_CAP" ]; then
    echo "No capacity found for $MODEL_NAME v$MODEL_VERSION ($SKU)"
    echo "Try a different SKU or version."
    exit 0
fi

echo "Capacity: $MODEL_NAME v$MODEL_VERSION ($SKU)"
echo ""
printf "%-22s %-12s %-15s %s\n" "Region" "Available" "Quota" "SKU"
printf -- '-%.0s' {1..60}; echo ""

for region in $REGIONS_WITH_CAP; do
    avail=$(echo "$CAPACITY_RESULT" | jq -r ".value[] | select(.location==\"$region\" and .properties.skuName==\"$SKU\") | .properties.availableCapacity" 2>/dev/null | head -1)

    # Check subscription quota
    usage_json=$(az cognitiveservices usage list --location "$region" --subscription "$SUB_ID" -o json 2>/dev/null || echo "[]")
    quota_avail=$(echo "$usage_json" | jq -r --arg name "OpenAI.$SKU.$MODEL_NAME" \
        '[.[] | select(.name.value == $name)] | if length > 0 then .[0].limit - .[0].currentValue else 0 end' 2>/dev/null || echo "?")

    if [ "$quota_avail" = "0" ]; then
        quota_display="0 (none)"
    elif [ "$quota_avail" = "?" ]; then
        quota_display="?"
    else
        quota_display="${quota_avail}K"
    fi

    printf "%-22s %-12s %-15s %s\n" "$region" "${avail}K TPM" "$quota_display" "$SKU"
done

# customize Examples

## Example 1: Basic Deployment with Defaults

**Scenario:** Deploy gpt-4o accepting all defaults for quick setup.
**Config:** gpt-4o / GlobalStandard / 10K TPM / Dynamic Quota enabled
**Result:** Deployment `gpt-4o` created in ~2-3 min with auto-upgrade enabled.

## Example 2: Production Deployment with Custom Capacity

**Scenario:** Deploy gpt-4o for production with high throughput.
**Config:** gpt-4o / GlobalStandard / 50K TPM / Dynamic Quota / Name: `gpt-4o-production`
**Result:** 50K TPM (500 req/10s). Suitable for moderate-to-high traffic production apps.

## Example 3: PTU Deployment for High-Volume Workload

**Scenario:** Deploy gpt-4o with reserved capacity (PTU) for predictable workload.
**Config:** gpt-4o / ProvisionedManaged / 200 PTU (min 50, max 1000) / Priority Processing enabled
**PTU sizing:** 40K input + 20K output tokens/min → ~100 PTU estimated → 200 PTU recommended (2x headroom)
**Result:** Guaranteed throughput, fixed monthly cost. Use case: customer service bots, document pipelines.

## Example 4: Development Deployment with Standard SKU

**Scenario:** Deploy gpt-4o-mini for dev/testing with minimal cost.
**Config:** gpt-4o-mini / Standard / 1K TPM / Name: `gpt-4o-mini-dev`
**Result:** 1K TPM, 10 req/10s. Minimal pay-per-use cost for development and prototyping.

## Example 5: Spillover Configuration

**Scenario:** Deploy gpt-4o with spillover to handle peak load overflow.
**Config:** gpt-4o / GlobalStandard / 20K TPM / Dynamic Quota / Spillover → `gpt-4o-backup`
**Result:** Primary handles up to 20K TPM; overflow auto-redirects to backup deployment.

## Example 6: Anthropic Model Deployment (claude-sonnet-4-6)

**Scenario:** Deploy claude-sonnet-4-6 with customized settings.
**Config:** claude-sonnet-4-6 / GlobalStandard / capacity 1 (MaaS) / Industry: Healthcare / No RAI policy (Anthropic manages content filtering)
**Result:** User selected "Healthcare" as industry → tenant country code (US) and org name fetched automatically → deployed via ARM REST API with `modelProviderData` in ~2 min.

---

## Comparison Matrix

| Scenario | Model | SKU | Capacity | Dynamic Quota | Priority | Spillover | Use Case |
|----------|-------|-----|----------|:---:|:---:|:---:|----------|
| Ex 1 | gpt-4o | GlobalStandard | 10K TPM | ✓ | - | - | Quick setup |
| Ex 2 | gpt-4o | GlobalStandard | 50K TPM | ✓ | - | - | Production |
| Ex 3 | gpt-4o | ProvisionedManaged | 200 PTU | - | ✓ | - | Predictable workload |
| Ex 4 | gpt-4o-mini | Standard | 1K TPM | - | - | - | Dev/testing |
| Ex 5 | gpt-4o | GlobalStandard | 20K TPM | ✓ | - | ✓ | Peak load |
| Ex 6 | claude-sonnet-4-6 | GlobalStandard | 1 (MaaS) | - | - | - | Anthropic model |

## Common Patterns

### Dev → Staging → Production

| Stage | Model | SKU | Capacity | Extras |
|-------|-------|-----|----------|--------|
| Dev | gpt-4o-mini | Standard | 1K TPM | — |
| Staging | gpt-4o | GlobalStandard | 10K TPM | — |
| Production | gpt-4o | GlobalStandard | 50K TPM | Dynamic Quota + Spillover |

### Cost Optimization

- **High priority:** gpt-4o, ProvisionedManaged, 100 PTU, Priority Processing
- **Low priority:** gpt-4o-mini, Standard, 5K TPM

---

## Tips and Best Practices

**Capacity:** Start conservative → monitor with Azure Monitor → scale gradually → use spillover for peaks.

**SKU Selection:** Standard for dev → GlobalStandard + dynamic quota for variable production → ProvisionedManaged (PTU) for predictable load.

**Cost:** Right-size capacity; use gpt-4o-mini where possible (80-90% accuracy at lower cost); enable dynamic quota; consider PTU for consistent high-volume.

**Versions:** Auto-upgrade recommended; test new versions in staging first; pin only if compatibility requires it.

**Content Filtering:** Start with DefaultV2; use custom policies only for specific needs; monitor filtered requests.

---

## Troubleshooting

| Problem | Solution |
|---------|----------|
| `QuotaExceeded` | Check usage with `az cognitiveservices usage list`, reduce capacity, try different SKU, check other regions, or use the [quota skill](../../../quota/quota.md) to request an increase |
| Version not available for SKU | Check `az cognitiveservices account list-models --query "[?name=='gpt-4o'].version"`, use latest |
| Deployment name exists | Skill auto-generates unique name (e.g., `gpt-4o-2`), or specify custom name |

---
name: customize
description: "Interactive guided deployment flow for Azure OpenAI models with full customization control. Step-by-step selection of model version, SKU (GlobalStandard/Standard/ProvisionedManaged), capacity, RAI policy (content filter), and advanced options (dynamic quota, priority processing, spillover). USE FOR: custom deployment, customize model deployment, choose version, select SKU, set capacity, configure content filter, RAI policy, deployment options, detailed deployment, advanced deployment, PTU deployment, provisioned throughput. DO NOT USE FOR: quick deployment to optimal region (use preset)."
license: MIT
metadata:
  author: Microsoft
  version: "1.0.1"
---

# Customize Model Deployment

Interactive guided workflow for deploying Azure OpenAI models with full customization control over version, SKU, capacity, content filtering, and advanced options.

## Quick Reference

| Property | Description |
|----------|-------------|
| **Flow** | Interactive step-by-step guided deployment |
| **Customization** | Version, SKU, Capacity, RAI Policy, Advanced Options |
| **SKU Support** | GlobalStandard, Standard, ProvisionedManaged, DataZoneStandard |
| **Best For** | Precise control over deployment configuration |
| **Authentication** | Azure CLI (`az login`) |
| **Tools** | Azure CLI, MCP tools (optional) |

## When to Use This Skill

Use this skill when you need **precise control** over deployment configuration:

- ✅ **Choose specific model version** (not just latest)
- ✅ **Select deployment SKU** (GlobalStandard vs Standard vs PTU)
- ✅ **Set exact capacity** within available range
- ✅ **Configure content filtering** (RAI policy selection)
- ✅ **Enable advanced features** (dynamic quota, priority processing, spillover)
- ✅ **PTU deployments** (Provisioned Throughput Units)

**Alternative:** Use `preset` for quick deployment to the best available region with automatic configuration.

### Comparison: customize vs preset

| Feature | customize | preset |
|---------|---------------------|----------------------------|
| **Focus** | Full customization control | Optimal region selection |
| **Version Selection** | User chooses from available | Uses latest automatically |
| **SKU Selection** | User chooses (GlobalStandard/Standard/PTU) | GlobalStandard only |
| **Capacity** | User specifies exact value | Auto-calculated (50% of available) |
| **RAI Policy** | User selects from options | Default policy only |
| **Region** | Current region first, falls back to all regions if no capacity | Checks capacity across all regions upfront |
| **Use Case** | Precise deployment requirements | Quick deployment to best region |

## Prerequisites

- Azure subscription with Cognitive Services Contributor or Owner role
- Azure AI Foundry project resource ID (format: `/subscriptions/{sub}/resourceGroups/{rg}/providers/Microsoft.CognitiveServices/accounts/{account}/projects/{project}`)
- Azure CLI installed and authenticated (`az login`)
- Optional: Set `PROJECT_RESOURCE_ID` environment variable

## Workflow Overview

### Complete Flow (14 Phases)

```
1. Verify Authentication
2. Get Project Resource ID
3. Verify Project Exists
4. Get Model Name (if not provided)
5. List Model Versions → User Selects
6. List SKUs for Version → User Selects
7. Get Capacity Range → User Configures
   7b. If no capacity: Cross-Region Fallback → Query all regions → User selects region/project
8. List RAI Policies → User Selects
9. Configure Advanced Options (if applicable)
10. Configure Version Upgrade Policy
11. Generate Deployment Name
12. Review Configuration
13. Execute Deployment & Monitor
```

### Fast Path (Defaults)

If user accepts all defaults (latest version, GlobalStandard SKU, recommended capacity, default RAI policy, standard upgrade policy), deployment completes in ~5 interactions.

---

## Phase Summaries

> ⚠️ **MUST READ:** Before executing any phase, load [references/customize-workflow.md](references/customize-workflow.md) for the full scripts and implementation details. The summaries below describe *what* each phase does — the reference file contains the *how* (CLI commands, quota patterns, capacity formulas, cross-region fallback logic).

| Phase | Action | Key Details |
|-------|--------|-------------|
| **1. Verify Auth** | Check `az account show`; prompt `az login` if needed | Verify correct subscription is active |
| **2. Get Project ID** | Read `PROJECT_RESOURCE_ID` env var or prompt user | ARM resource ID format required |
| **3. Verify Project** | Parse resource ID, call `az cognitiveservices account show` | Extracts subscription, RG, account, project, region |
| **4. Get Model** | List models via `az cognitiveservices account list-models` | User selects from available or enters custom name |
| **5. Select Version** | Query versions for chosen model | Recommend latest; user picks from list |
| **6. Select SKU** | Query model catalog + subscription quota, show only deployable SKUs | ⚠️ Never hardcode SKU lists — always query live data |
| **7. Configure Capacity** | Query capacity API, validate min/max/step, user enters value | Cross-region fallback if no capacity in current region |
| **8. Select RAI Policy** | Present content filter options | Default: `Microsoft.DefaultV2` |
| **9. Advanced Options** | Dynamic quota (GlobalStandard), priority processing (PTU), spillover | SKU-dependent availability |
| **10. Upgrade Policy** | Choose: OnceNewDefaultVersionAvailable / OnceCurrentVersionExpired / NoAutoUpgrade | Default: auto-upgrade on new default |
| **11. Deployment Name** | Auto-generate unique name, allow custom override | Validates format: `^[\w.-]{2,64}$` |
| **12. Review** | Display full config summary, confirm before proceeding | User approves or cancels |
| **13. Deploy & Monitor** | `az cognitiveservices account deployment create`, poll status | Timeout after 5 min; show endpoint + portal link |


---

## Error Handling

### Common Issues and Resolutions

| Error | Cause | Resolution |
|-------|-------|------------|
| **Model not found** | Invalid model name | List available models with `az cognitiveservices account list-models` |
| **Version not available** | Version not supported for SKU | Select different version or SKU |
| **Insufficient quota** | Capacity > available quota | Skill auto-searches all regions; fails only if no region has quota |
| **SKU not supported** | SKU not available in region | Cross-region fallback searches other regions automatically |
| **Capacity out of range** | Invalid capacity value | **PREVENTED**: Skill validates min/max/step at input (Phase 7) |
| **Deployment name exists** | Name conflict | Auto-incremented name generation |
| **Authentication failed** | Not logged in | Run `az login` |
| **Permission denied** | Insufficient permissions | Assign Cognitive Services Contributor role |
| **Capacity query fails** | API/permissions/network error | **DEPLOYMENT BLOCKED**: Will not proceed without valid quota data |

### Troubleshooting Commands

```bash
# Check deployment status
az cognitiveservices account deployment show --name <account> --resource-group <rg> --deployment-name <name>

# List all deployments
az cognitiveservices account deployment list --name <account> --resource-group <rg> -o table

# Check quota usage
az cognitiveservices usage list --name <account> --resource-group <rg>

# Delete failed deployment
az cognitiveservices account deployment delete --name <account> --resource-group <rg> --deployment-name <name>
```

---

## Selection Guides & Advanced Topics

> For SKU comparison tables, PTU sizing formulas, and advanced option details, load [references/customize-guides.md](references/customize-guides.md).

**SKU selection:** GlobalStandard (production/HA) → Standard (dev/test) → ProvisionedManaged (high-volume/guaranteed throughput) → DataZoneStandard (data residency).

**Capacity:** TPM-based SKUs range from 1K (dev) to 100K+ (large production). PTU-based use formula: `(Input TPM × 0.001) + (Output TPM × 0.002) + (Requests/min × 0.1)`.

**Advanced options:** Dynamic quota (GlobalStandard only), priority processing (PTU only, extra cost), spillover (overflow to backup deployment).

---

## Related Skills

- **preset** - Quick deployment to best region with automatic configuration
- **microsoft-foundry** - Parent skill for all Azure AI Foundry operations
- **[quota](../../../quota/quota.md)** — For quota viewing, increase requests, and troubleshooting quota errors, defer to this skill instead of duplicating guidance
- **rbac** - Manage permissions and access control

---

## Notes

- Set `PROJECT_RESOURCE_ID` environment variable to skip prompt
- Not all SKUs available in all regions; capacity varies by subscription/region/model
- Custom RAI policies can be configured in Azure Portal
- Automatic version upgrades occur during maintenance windows
- Use Azure Monitor and Application Insights for production deployments

# Customize Guides — Selection Guides & Advanced Topics

> Reference for: `models/deploy-model/customize/SKILL.md`

**Table of Contents:** [Selection Guides](#selection-guides) · [Advanced Topics](#advanced-topics)

## Selection Guides

### How to Choose SKU

| SKU | Best For | Cost | Availability |
|-----|----------|------|--------------|
| **GlobalStandard** | Production, high availability | Medium | Multi-region |
| **Standard** | Development, testing | Low | Single region |
| **ProvisionedManaged** | High-volume, predictable workloads | Fixed (PTU) | Reserved capacity |
| **DataZoneStandard** | Data residency requirements | Medium | Specific zones |

**Decision Tree:**
```
Do you need guaranteed throughput?
├─ Yes → ProvisionedManaged (PTU)
└─ No → Do you need high availability?
        ├─ Yes → GlobalStandard
        └─ No → Standard
```

### How to Choose Capacity

**For TPM-based SKUs (GlobalStandard, Standard):**

| Workload | Recommended Capacity |
|----------|---------------------|
| Development/Testing | 1K - 5K TPM |
| Small Production | 5K - 20K TPM |
| Medium Production | 20K - 100K TPM |
| Large Production | 100K+ TPM |

**For PTU-based SKUs (ProvisionedManaged):**

Use the PTU calculator based on:
- Input tokens per minute
- Output tokens per minute
- Requests per minute

**Capacity Planning Tips:**
- Start with recommended capacity
- Monitor usage and adjust
- Enable dynamic quota for flexibility
- Consider spillover for peak loads

### How to Choose RAI Policy

| Policy | Filtering Level | Use Case |
|--------|----------------|----------|
| **Microsoft.DefaultV2** | Balanced | Most applications |
| **Microsoft.Prompt-Shield** | Enhanced | Security-sensitive apps |
| **Custom** | Configurable | Specific requirements |

**Recommendation:** Start with `Microsoft.DefaultV2` and adjust based on application needs.

---

## Advanced Topics

### PTU (Provisioned Throughput Units) Deployments

**What is PTU?**
- Reserved capacity with guaranteed throughput
- Measured in PTU units, not TPM
- Fixed cost regardless of usage
- Best for high-volume, predictable workloads

**PTU Calculator:**

```
Estimated PTU = (Input TPM × 0.001) + (Output TPM × 0.002) + (Requests/min × 0.1)

Example:
- Input: 10,000 tokens/min
- Output: 5,000 tokens/min
- Requests: 100/min

PTU = (10,000 × 0.001) + (5,000 × 0.002) + (100 × 0.1)
    = 10 + 10 + 10
    = 30 PTU
```

**PTU Deployment:**
```bash
az cognitiveservices account deployment create \
  --name <account-name> \
  --resource-group <resource-group> \
  --deployment-name <deployment-name> \
  --model-name <model-name> \
  --model-version <version> \
  --model-format "OpenAI" \
  --sku-name "ProvisionedManaged" \
  --sku-capacity 100  # PTU units
```

### Spillover Configuration

**Spillover Workflow:**
1. Primary deployment receives requests
2. When capacity reached, requests overflow to spillover target
3. Spillover target must be same model or compatible
4. Configure via deployment properties

**Best Practices:**
- Use spillover for peak load handling
- Spillover target should have sufficient capacity
- Monitor both deployments
- Test failover behavior

### Priority Processing

**What is Priority Processing?**
- Prioritizes your requests during high load
- Available for ProvisionedManaged SKU
- Additional charges apply
- Ensures consistent performance

**When to Use:**
- Mission-critical applications
- SLA requirements
- High-concurrency scenarios

# Customize Workflow — Detailed Phase Instructions

> Reference for: `models/deploy-model/customize/SKILL.md`

## Phase 1: Verify Authentication

```bash
az account show --query "{Subscription:name, User:user.name}" -o table
```

If not logged in: `az login`

Set subscription if needed:
```bash
az account list --query "[].[name,id,state]" -o table
az account set --subscription <subscription-id>
```

---

## Phase 2: Get Project Resource ID

Check `PROJECT_RESOURCE_ID` env var. If not set, prompt user.

**Format:** `/subscriptions/{sub}/resourceGroups/{rg}/providers/Microsoft.CognitiveServices/accounts/{account}/projects/{project}`

---

## Phase 3: Parse and Verify Project

Parse ARM resource ID to extract components:

```powershell
$SUBSCRIPTION_ID = ($PROJECT_RESOURCE_ID -split '/')[2]
$RESOURCE_GROUP = ($PROJECT_RESOURCE_ID -split '/')[4]
$ACCOUNT_NAME = ($PROJECT_RESOURCE_ID -split '/')[8]
$PROJECT_NAME = ($PROJECT_RESOURCE_ID -split '/')[10]
```

Verify project exists and get region:
```bash
az account set --subscription $SUBSCRIPTION_ID
az cognitiveservices account show \
  --name $ACCOUNT_NAME \
  --resource-group $RESOURCE_GROUP \
  --query location -o tsv
```

---

## Phase 4: Get Model Name

List available models if not provided:
```bash
az cognitiveservices account list-models \
  --name $ACCOUNT_NAME \
  --resource-group $RESOURCE_GROUP \
  --query "[].name" -o json
```

Present sorted unique list. Allow custom model name entry.

**Detect model format:**

```bash
# Get model format (e.g., OpenAI, Anthropic, Meta-Llama, Mistral, Cohere)
MODEL_FORMAT=$(az cognitiveservices account list-models \
  --name "$ACCOUNT_NAME" \
  --resource-group "$RESOURCE_GROUP" \
  --query "[?name=='$MODEL_NAME'].format" -o tsv | head -1)

MODEL_FORMAT=${MODEL_FORMAT:-"OpenAI"}
echo "Model format: $MODEL_FORMAT"
```

> 💡 **Model format determines the deployment path:**
> - `OpenAI` — Standard CLI, TPM-based capacity, RAI policies, version upgrade policies
> - `Anthropic` — REST API with `modelProviderData`, capacity=1, no RAI, no version upgrade
> - All other formats (`Meta-Llama`, `Mistral`, `Cohere`, etc.) — Standard CLI, capacity=1 (MaaS), no RAI, no version upgrade

---

## Phase 5: List and Select Model Version

```bash
az cognitiveservices account list-models \
  --name $ACCOUNT_NAME \
  --resource-group $RESOURCE_GROUP \
  --query "[?name=='$MODEL_NAME'].version" -o json
```

Recommend latest version (first in list). Default to `"latest"` if no versions found.

---

## Phase 6: List and Select SKU

> ⚠️ **Warning:** Never hardcode SKU lists — always query live data.

**Step A — Query model-supported SKUs:**
```bash
az cognitiveservices model list \
  --location $PROJECT_REGION \
  --subscription $SUBSCRIPTION_ID -o json
```

Filter: `model.name == $MODEL_NAME && model.version == $MODEL_VERSION`, extract `model.skus[].name`.

**Step B — Check subscription quota per SKU:**
```bash
az cognitiveservices usage list \
  --location $PROJECT_REGION \
  --subscription $SUBSCRIPTION_ID -o json
```

Quota key pattern: `OpenAI.<SKU>.<model-name>`. Calculate `available = limit - currentValue`.

**Step C — Present only deployable SKUs** (available > 0). If no SKUs have quota, direct user to the [quota skill](../../../../quota/quota.md).

---

## Phase 7: Configure Capacity

> ⚠️ **Non-OpenAI models (MaaS):** If `MODEL_FORMAT != "OpenAI"`, capacity is always `1` (pay-per-token billing). Skip capacity configuration and set `DEPLOY_CAPACITY=1`. Proceed to Phase 7c (Anthropic) or Phase 8.

**For OpenAI models only — query capacity via REST API:**
```bash
# Current region capacity
az rest --method GET --url \
  "https://management.azure.com/subscriptions/$SUBSCRIPTION_ID/providers/Microsoft.CognitiveServices/locations/$PROJECT_REGION/modelCapacities?api-version=2024-10-01&modelFormat=$MODEL_FORMAT&modelName=$MODEL_NAME&modelVersion=$MODEL_VERSION"
```

Filter result for `properties.skuName == $SELECTED_SKU`. Read `properties.availableCapacity`.

**Capacity defaults by SKU (OpenAI only):**

| SKU | Unit | Min | Max | Step | Default |
|-----|------|-----|-----|------|---------|
| ProvisionedManaged | PTU | 50 | 1000 | 50 | 100 |
| Others (TPM-based) | TPM | 1000 | min(available, 300000) | 1000 | min(10000, available/2) |

Validate user input: must be >= min, <= max, multiple of step. On invalid input, explain constraints.

### Phase 7b: Cross-Region Fallback

If no capacity in current region, query ALL regions:
```bash
az rest --method GET --url \
  "https://management.azure.com/subscriptions/$SUBSCRIPTION_ID/providers/Microsoft.CognitiveServices/modelCapacities?api-version=2024-10-01&modelFormat=$MODEL_FORMAT&modelName=$MODEL_NAME&modelVersion=$MODEL_VERSION"
```

Filter: `properties.skuName == $SELECTED_SKU && properties.availableCapacity > 0`. Sort descending by capacity.

Present available regions. After user selects region, find existing projects there:
```bash
az cognitiveservices account list \
  --query "[?kind=='AIProject' && location=='$PROJECT_REGION'].{Name:name, ResourceGroup:resourceGroup}" \
  -o json
```

If projects exist, let user select one and update `$ACCOUNT_NAME`, `$RESOURCE_GROUP`. If none, direct to project/create skill.

Re-run capacity configuration with new region's available capacity.

If no region has capacity: fail with guidance to request quota increase, check existing deployments, or try different model/SKU.

---

## Phase 7c: Anthropic Model Provider Data (Anthropic models only)

> ⚠️ **Only execute this phase if `MODEL_FORMAT == "Anthropic"`.** For OpenAI and other models, skip to Phase 8.

Anthropic models require `modelProviderData` in the deployment payload. Collect this before deployment.

**Step 1: Prompt user to select industry**

Present the following list and ask the user to choose one:

```
 1. None                    (API value: none)
 2. Biotechnology           (API value: biotechnology)
 3. Consulting              (API value: consulting)
 4. Education               (API value: education)
 5. Finance                 (API value: finance)
 6. Food & Beverage         (API value: food_and_beverage)
 7. Government              (API value: government)
 8. Healthcare              (API value: healthcare)
 9. Insurance               (API value: insurance)
10. Law                     (API value: law)
11. Manufacturing           (API value: manufacturing)
12. Media                   (API value: media)
13. Nonprofit               (API value: nonprofit)
14. Technology              (API value: technology)
15. Telecommunications      (API value: telecommunications)
16. Sport & Recreation      (API value: sport_and_recreation)
17. Real Estate             (API value: real_estate)
18. Retail                  (API value: retail)
19. Other                   (API value: other)
```

> ⚠️ **Do NOT pick a default industry or hardcode a value. Always ask the user.** This is required by Anthropic's terms of service. The industry list is static — there is no REST API that provides it.

Store selection as `SELECTED_INDUSTRY` (use the API value, e.g., `technology`).

**Step 2: Fetch tenant info (country code and organization name)**

```bash
TENANT_INFO=$(az rest --method GET \
  --url "https://management.azure.com/tenants?api-version=2024-11-01" \
  --query "value[0].{countryCode:countryCode, displayName:displayName}" -o json)

COUNTRY_CODE=$(echo "$TENANT_INFO" | jq -r '.countryCode')
ORG_NAME=$(echo "$TENANT_INFO" | jq -r '.displayName')
```

*PowerShell version:*
```powershell
$tenantInfo = az rest --method GET `
  --url "https://management.azure.com/tenants?api-version=2024-11-01" `
  --query "value[0].{countryCode:countryCode, displayName:displayName}" -o json | ConvertFrom-Json

$countryCode = $tenantInfo.countryCode
$orgName = $tenantInfo.displayName
```

Store `COUNTRY_CODE` and `ORG_NAME` for use in Phase 13.

---

## Phase 8: Select RAI Policy (Content Filter)

> ⚠️ **Note:** RAI policies only apply to OpenAI models. Skip this phase if `MODEL_FORMAT != "OpenAI"` (Anthropic, Meta-Llama, Mistral, Cohere, etc. do not use RAI policies).

Present options:
1. `Microsoft.DefaultV2` — Balanced filtering (recommended). Filters hate, violence, sexual, self-harm.
2. `Microsoft.Prompt-Shield` — Enhanced prompt injection/jailbreak protection.
3. Custom policies — Organization-specific (configured in Azure Portal).

Default: `Microsoft.DefaultV2`.

---

## Phase 9: Configure Advanced Options

Options are SKU-dependent:

**A. Dynamic Quota** (GlobalStandard only)
- Auto-scales beyond base allocation when capacity available
- Default: enabled

**B. Priority Processing** (ProvisionedManaged only)
- Prioritizes requests during high load; additional charges apply
- Default: disabled

**C. Spillover** (any SKU)
- Redirects requests to backup deployment at capacity
- Requires existing deployment; list with:
```bash
az cognitiveservices account deployment list \
  --name $ACCOUNT_NAME \
  --resource-group $RESOURCE_GROUP \
  --query "[].name" -o json
```
- Default: disabled

---

## Phase 10: Configure Version Upgrade Policy

> ⚠️ **Note:** Version upgrade policies only apply to OpenAI models. Skip this phase if `MODEL_FORMAT != "OpenAI"`.

| Policy | Description |
|--------|-------------|
| `OnceNewDefaultVersionAvailable` | Auto-upgrade to new default (Recommended) |
| `OnceCurrentVersionExpired` | Upgrade only when current expires |
| `NoAutoUpgrade` | Manual upgrade only |

Default: `OnceNewDefaultVersionAvailable`.

---

## Phase 11: Generate Deployment Name

List existing deployments to avoid conflicts:
```bash
az cognitiveservices account deployment list \
  --name $ACCOUNT_NAME \
  --resource-group $RESOURCE_GROUP \
  --query "[].name" -o json
```

Auto-generate: use model name as base, append `-2`, `-3` etc. if taken. Allow custom override. Validate: `^[\w.-]{2,64}$`.

---

## Phase 12: Review Configuration

Display summary of all selections for user confirmation before proceeding:
- Model, version, deployment name
- SKU, capacity (with unit), region
- RAI policy, version upgrade policy
- Advanced options (dynamic quota, priority, spillover)
- Account, resource group, project

User confirms or cancels.

---

## Phase 13: Execute Deployment

> 💡 `MODEL_FORMAT` was already detected in Phase 4. Use the stored value here.

### Standard CLI deployment (non-Anthropic models):

**Create deployment:**
```bash
az cognitiveservices account deployment create \
  --name $ACCOUNT_NAME \
  --resource-group $RESOURCE_GROUP \
  --deployment-name $DEPLOYMENT_NAME \
  --model-name $MODEL_NAME \
  --model-version $MODEL_VERSION \
  --model-format "$MODEL_FORMAT" \
  --sku-name $SELECTED_SKU \
  --sku-capacity $DEPLOY_CAPACITY
```

> 💡 **Note:** For non-OpenAI MaaS models, `$DEPLOY_CAPACITY` is `1` (set in Phase 7).

### Anthropic model deployment (requires modelProviderData):

The Azure CLI does not support `--model-provider-data`. Use the ARM REST API directly.

> ⚠️ Industry, country code, and organization name should have been collected in Phase 7c.

```bash
echo "Creating Anthropic model deployment via REST API..."

az rest --method PUT \
  --url "https://management.azure.com/subscriptions/$SUBSCRIPTION_ID/resourceGroups/$RESOURCE_GROUP/providers/Microsoft.CognitiveServices/accounts/$ACCOUNT_NAME/deployments/$DEPLOYMENT_NAME?api-version=2024-10-01" \
  --body "{
    \"sku\": {
      \"name\": \"$SELECTED_SKU\",
      \"capacity\": 1
    },
    \"properties\": {
      \"model\": {
        \"format\": \"Anthropic\",
        \"name\": \"$MODEL_NAME\",
        \"version\": \"$MODEL_VERSION\"
      },
      \"modelProviderData\": {
        \"industry\": \"$SELECTED_INDUSTRY\",
        \"countryCode\": \"$COUNTRY_CODE\",
        \"organizationName\": \"$ORG_NAME\"
      }
    }
  }"
```

*PowerShell version:*
```powershell
Write-Host "Creating Anthropic model deployment via REST API..."

$body = @{
    sku = @{
        name = $SELECTED_SKU
        capacity = 1
    }
    properties = @{
        model = @{
            format = "Anthropic"
            name = $MODEL_NAME
            version = $MODEL_VERSION
        }
        modelProviderData = @{
            industry = $SELECTED_INDUSTRY
            countryCode = $countryCode
            organizationName = $orgName
        }
    }
} | ConvertTo-Json -Depth 5

az rest --method PUT `
  --url "https://management.azure.com/subscriptions/$SUBSCRIPTION_ID/resourceGroups/$RESOURCE_GROUP/providers/Microsoft.CognitiveServices/accounts/$ACCOUNT_NAME/deployments/${DEPLOYMENT_NAME}?api-version=2024-10-01" `
  --body $body
```

> 💡 **Note:** Anthropic models use `capacity: 1` (MaaS billing model), not TPM-based capacity. RAI policy is not applicable for Anthropic models.

### Monitor deployment status:
```bash
az cognitiveservices account deployment show \
  --name $ACCOUNT_NAME \
  --resource-group $RESOURCE_GROUP \
  --deployment-name $DEPLOYMENT_NAME \
  --query "properties.provisioningState" -o tsv
```

Poll until `Succeeded` or `Failed`. Timeout after 5 minutes.

**Get endpoint:**
```bash
az cognitiveservices account show \
  --name $ACCOUNT_NAME \
  --resource-group $RESOURCE_GROUP \
  --query "properties.endpoint" -o tsv
```

On success, display deployment name, model, version, SKU, capacity, region, RAI policy, rate limits, endpoint, and Azure AI Foundry portal link.

# Examples: preset

## Example 1: Fast Path — Current Region Has Capacity

**Scenario:** Deploy gpt-4o to project in East US, which has capacity.
**Result:** Deployed in ~45s. No region selection needed. 100K TPM default, GlobalStandard SKU.

## Example 2: Alternative Region — No Capacity in Current Region

**Scenario:** Deploy gpt-4-turbo to dev project in West US 2 (no capacity).
**Result:** Queried all regions → user selected East US 2 (120K available) → deployed in ~2 min.

## Example 3: Create New Project in Optimal Region

**Scenario:** Deploy gpt-4o-mini in Europe for data residency; no existing European project.
**Result:** Created AI Services hub + project in Sweden Central → deployed in ~4 min with 150K TPM.

## Example 4: Insufficient Quota Everywhere

**Scenario:** Deploy gpt-4 but all regions have exhausted quota.
**Result:** Graceful failure with actionable guidance:
1. Request quota increase via the [quota skill](../../../quota/quota.md)
2. List existing deployments consuming quota
3. Suggest alternative models (gpt-4o, gpt-4o-mini)

## Example 5: First-Time User — No Project

**Scenario:** Deploy gpt-4o with no existing AI Foundry project.
**Result:** Full onboarding in ~5 min — created resource group, AI Services hub, project, then deployed.

## Example 6: Deployment Name Conflict

**Scenario:** Auto-generated deployment name already exists.
**Result:** Appended random hex suffix (e.g., `-7b9e`) and retried automatically.

## Example 7: Multi-Version Model Selection

**Scenario:** Deploy "latest gpt-4o" when multiple versions exist.
**Result:** Latest stable version auto-selected. Capacity aggregated across versions.

## Example 8: Anthropic Model (claude-sonnet-4-6)

**Scenario:** Deploy claude-sonnet-4-6 (Anthropic model requiring modelProviderData).
**Result:** User prompted for industry selection → tenant country code and org name fetched automatically → deployed via ARM REST API with `modelProviderData` payload in ~2 min. Capacity set to 1 (MaaS billing).

---

## Summary of Scenarios

| Scenario | Duration | Key Features |
|----------|----------|--------------|
| **1: Fast Path** | ~45s | Current region has capacity, direct deploy |
| **2: Alt Region** | ~2m | Region selection, project switch |
| **3: New Project** | ~4m | Project creation in optimal region |
| **4: No Quota** | N/A | Graceful failure, actionable guidance |
| **5: First-Time** | ~5m | Complete onboarding |
| **6: Name Conflict** | ~1m | Auto-retry with suffix |
| **7: Multi-Version** | ~1m | Latest version auto-selected |
| **8: Anthropic** | ~2m | Industry prompt, tenant info, REST API deploy |

## Common Patterns

```
A: Quick Deploy     Auth → Get Project → Check Region (✓) → Deploy
B: Region Select    Auth → Get Project → Region (✗) → Query All → Select → Deploy
C: Full Onboarding  Auth → No Projects → Create Project → Deploy
D: Error Recovery   Deploy (✗) → Analyze → Fix → Retry
```

---
name: preset
description: "Intelligently deploys Azure OpenAI models to optimal regions by analyzing capacity across all available regions. Automatically checks current region first and shows alternatives if needed. USE FOR: quick deployment, optimal region, best region, automatic region selection, fast setup, multi-region capacity check, high availability deployment, deploy to best location. DO NOT USE FOR: custom SKU selection (use customize), specific version selection (use customize), custom capacity configuration (use customize), PTU deployments (use customize)."
license: MIT
metadata:
  author: Microsoft
  version: "1.0.1"
---

# Deploy Model to Optimal Region

Automates intelligent Azure OpenAI model deployment by checking capacity across regions and deploying to the best available option.

## What This Skill Does

1. Verifies Azure authentication and project scope
2. Checks capacity in current project's region
3. If no capacity: analyzes all regions and shows available alternatives
4. Filters projects by selected region
5. Supports creating new projects if needed
6. Deploys model with GlobalStandard SKU
7. Monitors deployment progress

## Prerequisites

- Azure CLI installed and configured
- Active Azure subscription with Cognitive Services read/create permissions
- Azure AI Foundry project resource ID (`PROJECT_RESOURCE_ID` env var or provided interactively)
  - Format: `/subscriptions/{sub-id}/resourceGroups/{rg}/providers/Microsoft.CognitiveServices/accounts/{account}/projects/{project}`
  - Found in: Azure AI Foundry portal → Project → Overview → Resource ID

## Quick Workflow

### Fast Path (Current Region Has Capacity)
```
1. Check authentication → 2. Get project → 3. Check current region capacity
→ 4. Deploy immediately
```

### Alternative Region Path (No Capacity)
```
1. Check authentication → 2. Get project → 3. Check current region (no capacity)
→ 4. Query all regions → 5. Show alternatives → 6. Select region + project
→ 7. Deploy
```

---

## Deployment Phases

| Phase | Action | Key Commands |
|-------|--------|-------------|
| 1. Verify Auth | Check Azure CLI login and subscription | `az account show`, `az login` |
| 2. Get Project | Parse `PROJECT_RESOURCE_ID` ARM ID, verify exists | `az cognitiveservices account show` |
| 3. Get Model | List available models, user selects model + version | `az cognitiveservices account list-models` |
| 4. Check Current Region | Query capacity using GlobalStandard SKU | `az rest --method GET .../modelCapacities` |
| 5. Multi-Region Query | If no local capacity, query all regions | Same capacity API without location filter |
| 6. Select Region + Project | User picks region; find or create project | `az cognitiveservices account list`, `az cognitiveservices account create` |
| 7. Deploy | Generate unique name, calculate capacity (50% available, min 50 TPM), create deployment | `az cognitiveservices account deployment create` |

For detailed step-by-step instructions, see [workflow reference](references/workflow.md).

---

## Error Handling

| Error | Symptom | Resolution |
|-------|---------|------------|
| Auth failure | `az account show` returns error | Run `az login` then `az account set --subscription <id>` |
| No quota | All regions show 0 capacity | Defer to the [quota skill](../../../quota/quota.md) for increase requests and troubleshooting; check existing deployments; try alternative models |
| Model not found | Empty capacity list | Verify model name with `az cognitiveservices account list-models`; check case sensitivity |
| Name conflict | "deployment already exists" | Append suffix to deployment name (handled automatically by `generate_deployment_name` script) |
| Region unavailable | Region doesn't support model | Select a different region from the available list |
| Permission denied | "Forbidden" or "Unauthorized" | Verify Cognitive Services Contributor role: `az role assignment list --assignee <user>` |

---

## Advanced Usage

```bash
# Custom capacity
az cognitiveservices account deployment create ... --sku-capacity <value>

# Check deployment status
az cognitiveservices account deployment show --name <acct> --resource-group <rg> --deployment-name <name> --query "{Status:properties.provisioningState}"

# Delete deployment
az cognitiveservices account deployment delete --name <acct> --resource-group <rg> --deployment-name <name>
```

## Notes

- **SKU:** GlobalStandard only — **API Version:** 2024-10-01 (GA stable)

---

## Related Skills

- **microsoft-foundry** - Parent skill for Azure AI Foundry operations
- **[quota](../../../quota/quota.md)** — For quota viewing, increase requests, and troubleshooting quota errors, defer to this skill
- **azure-quick-review** - Review Azure resources for compliance
- **azure-cost-estimation** - Estimate costs for Azure deployments
- **azure-validate** - Validate Azure infrastructure before deployment

# Preset Deployment Workflow - Detailed Implementation

This file contains the full step-by-step bash/PowerShell scripts for preset (optimal region) model deployment. Referenced from the main [SKILL.md](../SKILL.md).

---

## Phase 1: Verify Authentication

Check if user is logged into Azure CLI:

```bash
az account show --query "{Subscription:name, User:user.name}" -o table
```

**If not logged in:**
```bash
az login
```

**Verify subscription is correct:**
```bash
# List all subscriptions
az account list --query "[].[name,id,state]" -o table

# Set active subscription if needed
az account set --subscription <subscription-id>
```

---

## Phase 2: Get Current Project

**Check for PROJECT_RESOURCE_ID environment variable first:**

```bash
if [ -n "$PROJECT_RESOURCE_ID" ]; then
  echo "Using project resource ID from environment: $PROJECT_RESOURCE_ID"
else
  echo "PROJECT_RESOURCE_ID not set. Please provide your Azure AI Foundry project resource ID."
  echo ""
  echo "You can find this in:"
  echo "  • Azure AI Foundry portal → Project → Overview → Resource ID"
  echo "  • Format: /subscriptions/{sub-id}/resourceGroups/{rg}/providers/Microsoft.CognitiveServices/accounts/{account}/projects/{project}"
  echo ""
  echo "Example: /subscriptions/abc123.../resourceGroups/rg-prod/providers/Microsoft.CognitiveServices/accounts/my-account/projects/my-project"
  echo ""
  read -p "Enter project resource ID: " PROJECT_RESOURCE_ID
fi
```

**Parse the ARM resource ID to extract components:**

```bash
# Extract components from ARM resource ID
# Format: /subscriptions/{sub-id}/resourceGroups/{rg}/providers/Microsoft.CognitiveServices/accounts/{account}/projects/{project}

SUBSCRIPTION_ID=$(echo "$PROJECT_RESOURCE_ID" | sed -n 's|.*/subscriptions/\([^/]*\).*|\1|p')
RESOURCE_GROUP=$(echo "$PROJECT_RESOURCE_ID" | sed -n 's|.*/resourceGroups/\([^/]*\).*|\1|p')
ACCOUNT_NAME=$(echo "$PROJECT_RESOURCE_ID" | sed -n 's|.*/accounts/\([^/]*\)/projects.*|\1|p')
PROJECT_NAME=$(echo "$PROJECT_RESOURCE_ID" | sed -n 's|.*/projects/\([^/?]*\).*|\1|p')

if [ -z "$SUBSCRIPTION_ID" ] || [ -z "$RESOURCE_GROUP" ] || [ -z "$ACCOUNT_NAME" ] || [ -z "$PROJECT_NAME" ]; then
  echo "❌ Invalid project resource ID format"
  echo "Expected format: /subscriptions/{sub-id}/resourceGroups/{rg}/providers/Microsoft.CognitiveServices/accounts/{account}/projects/{project}"
  exit 1
fi

echo "Parsed project details:"
echo "  Subscription: $SUBSCRIPTION_ID"
echo "  Resource Group: $RESOURCE_GROUP"
echo "  Account: $ACCOUNT_NAME"
echo "  Project: $PROJECT_NAME"
```

**Verify the project exists and get its region:**

```bash
# Set active subscription
az account set --subscription "$SUBSCRIPTION_ID"

# Get project details to verify it exists and extract region
PROJECT_REGION=$(az cognitiveservices account show \
  --name "$PROJECT_NAME" \
  --resource-group "$RESOURCE_GROUP" \
  --query location -o tsv 2>/dev/null)

if [ -z "$PROJECT_REGION" ]; then
  echo "❌ Project '$PROJECT_NAME' not found in resource group '$RESOURCE_GROUP'"
  echo ""
  echo "Please verify the resource ID is correct."
  echo ""
  echo "List available projects:"
  echo "  az cognitiveservices account list --query \"[?kind=='AIProject'].{Name:name, Location:location, ResourceGroup:resourceGroup}\" -o table"
  exit 1
fi

echo "✓ Project found"
echo "  Region: $PROJECT_REGION"
```

---

## Phase 3: Get Model Name

**If model name provided as skill parameter, skip this phase.**

Ask user which model to deploy. **Fetch available models dynamically** from the account rather than using a hardcoded list:

```bash
# List available models in the account
az cognitiveservices account list-models \
  --name "$PROJECT_NAME" \
  --resource-group "$RESOURCE_GROUP" \
  --query "[].name" -o tsv | sort -u
```

Present the results to the user and let them choose, or enter a custom model name.

**Store model:**
```bash
MODEL_NAME="<selected-model>"
```

**Get model version (latest stable):**
```bash
# List available models and versions in the account
az cognitiveservices account list-models \
  --name "$PROJECT_NAME" \
  --resource-group "$RESOURCE_GROUP" \
  --query "[?name=='$MODEL_NAME'].{Name:name, Version:version, Format:format}" \
  -o table
```

**Use latest version or let user specify:**
```bash
MODEL_VERSION="<version-or-latest>"
```

**Detect model format:**

```bash
# Get model format from model catalog (e.g., OpenAI, Anthropic, Meta-Llama, Mistral, Cohere)
MODEL_FORMAT=$(az cognitiveservices account list-models \
  --name "$ACCOUNT_NAME" \
  --resource-group "$RESOURCE_GROUP" \
  --query "[?name=='$MODEL_NAME'].format" -o tsv | head -1)

# Default to OpenAI if not found
MODEL_FORMAT=${MODEL_FORMAT:-"OpenAI"}

echo "Model format: $MODEL_FORMAT"
```

> 💡 **Model format determines the deployment path:**
> - `OpenAI` — Standard CLI deployment, TPM-based capacity, RAI policies apply
> - `Anthropic` — REST API deployment with `modelProviderData`, capacity=1, no RAI
> - All other formats (`Meta-Llama`, `Mistral`, `Cohere`, etc.) — Standard CLI deployment, capacity=1 (MaaS), no RAI

---

## Phase 4: Check Current Region Capacity

Before checking other regions, see if the current project's region has capacity:

```bash
# Query capacity for current region
CAPACITY_JSON=$(az rest --method GET \
  --url "https://management.azure.com/subscriptions/$SUBSCRIPTION_ID/providers/Microsoft.CognitiveServices/locations/$PROJECT_REGION/modelCapacities?api-version=2024-10-01&modelFormat=$MODEL_FORMAT&modelName=$MODEL_NAME&modelVersion=$MODEL_VERSION")

# Extract available capacity for GlobalStandard SKU
CURRENT_CAPACITY=$(echo "$CAPACITY_JSON" | jq -r '.value[] | select(.properties.skuName=="GlobalStandard") | .properties.availableCapacity')
```

**Check result:**
```bash
if [ -n "$CURRENT_CAPACITY" ] && [ "$CURRENT_CAPACITY" -gt 0 ]; then
  echo "✓ Current region ($PROJECT_REGION) has capacity: $CURRENT_CAPACITY TPM"
  echo "Proceeding with deployment..."
  # Skip to Phase 7 (Deploy)
else
  echo "⚠ Current region ($PROJECT_REGION) has no available capacity"
  echo "Checking alternative regions..."
  # Continue to Phase 5
fi
```

---

## Phase 5: Query Multi-Region Capacity (If Needed)

Only execute this phase if current region has no capacity.

**Query capacity across all regions:**
```bash
# Get capacity for all regions in subscription
ALL_REGIONS_JSON=$(az rest --method GET \
  --url "https://management.azure.com/subscriptions/$SUBSCRIPTION_ID/providers/Microsoft.CognitiveServices/modelCapacities?api-version=2024-10-01&modelFormat=$MODEL_FORMAT&modelName=$MODEL_NAME&modelVersion=$MODEL_VERSION")

# Save to file for processing
echo "$ALL_REGIONS_JSON" > /tmp/capacity_check.json
```

**Parse and categorize regions:**
```bash
# Extract available regions (capacity > 0)
AVAILABLE_REGIONS=$(jq -r '.value[] | select(.properties.skuName=="GlobalStandard" and .properties.availableCapacity > 0) | "\(.location)|\(.properties.availableCapacity)"' /tmp/capacity_check.json)

# Extract unavailable regions (capacity = 0 or undefined)
UNAVAILABLE_REGIONS=$(jq -r '.value[] | select(.properties.skuName=="GlobalStandard" and (.properties.availableCapacity == 0 or .properties.availableCapacity == null)) | "\(.location)|0"' /tmp/capacity_check.json)
```

**Format and display regions:**
```bash
# Format capacity (e.g., 120000 -> 120K)
format_capacity() {
  local capacity=$1
  if [ "$capacity" -ge 1000000 ]; then
    echo "$(awk "BEGIN {printf \"%.1f\", $capacity/1000000}")M TPM"
  elif [ "$capacity" -ge 1000 ]; then
    echo "$(awk "BEGIN {printf \"%.0f\", $capacity/1000}")K TPM"
  else
    echo "$capacity TPM"
  fi
}

echo ""
echo "⚠ No Capacity in Current Region"
echo ""
echo "The current project's region ($PROJECT_REGION) does not have available capacity for $MODEL_NAME."
echo ""
echo "Available Regions (with capacity):"
echo ""

# Display available regions with formatted capacity
echo "$AVAILABLE_REGIONS" | while IFS='|' read -r region capacity; do
  formatted_capacity=$(format_capacity "$capacity")
  # Get region display name (capitalize and format)
  region_display=$(echo "$region" | sed 's/\([a-z]\)\([a-z]*\)/\U\1\L\2/g; s/\([a-z]\)\([0-9]\)/\1 \2/g')
  echo "  • $region_display - $formatted_capacity"
done

echo ""
echo "Unavailable Regions:"
echo ""

# Display unavailable regions
echo "$UNAVAILABLE_REGIONS" | while IFS='|' read -r region capacity; do
  region_display=$(echo "$region" | sed 's/\([a-z]\)\([a-z]*\)/\U\1\L\2/g; s/\([a-z]\)\([0-9]\)/\1 \2/g')
  if [ "$capacity" = "0" ]; then
    echo "  ✗ $region_display (Insufficient quota - 0 TPM available)"
  else
    echo "  ✗ $region_display (Model not supported)"
  fi
done
```

**Handle no capacity anywhere:**
```bash
if [ -z "$AVAILABLE_REGIONS" ]; then
  echo ""
  echo "❌ No Available Capacity in Any Region"
  echo ""
  echo "No regions have available capacity for $MODEL_NAME with GlobalStandard SKU."
  echo ""
  echo "Next Steps:"
  echo "1. Request quota increase — use the quota skill (../../../quota/quota.md)"
  echo ""
  echo "2. Check existing deployments (may be using quota):"
  echo "   az cognitiveservices account deployment list \\"
  echo "     --name $PROJECT_NAME \\"
  echo "     --resource-group $RESOURCE_GROUP"
  echo ""
  echo "3. Consider alternative models with lower capacity requirements:"
  echo "   • gpt-4o-mini (cost-effective, lower capacity requirements)"
  echo "   List available models: az cognitiveservices account list-models --name \$PROJECT_NAME --resource-group \$RESOURCE_GROUP --output table"
  exit 1
fi
```

---

## Phase 6: Select Region and Project

**Ask user to select region from available options.**

Example using AskUserQuestion:
- Present available regions as options
- Show capacity for each
- User selects preferred region

**Store selection:**
```bash
SELECTED_REGION="<user-selected-region>"  # e.g., "eastus2"
```

**Find projects in selected region:**
```bash
PROJECTS_IN_REGION=$(az cognitiveservices account list \
  --query "[?kind=='AIProject' && location=='$SELECTED_REGION'].{Name:name, ResourceGroup:resourceGroup}" \
  --output json)

PROJECT_COUNT=$(echo "$PROJECTS_IN_REGION" | jq '. | length')

if [ "$PROJECT_COUNT" -eq 0 ]; then
  echo "No projects found in $SELECTED_REGION"
  echo "Would you like to create a new project? (yes/no)"
  # If yes, continue to project creation
  # If no, exit or select different region
else
  echo "Projects in $SELECTED_REGION:"
  echo "$PROJECTS_IN_REGION" | jq -r '.[] | "  • \(.Name) (\(.ResourceGroup))"'
  echo ""
  echo "Select a project or create new project"
fi
```

**Option A: Use existing project**
```bash
PROJECT_NAME="<selected-project-name>"
RESOURCE_GROUP="<resource-group>"
```

**Option B: Create new project**
```bash
# Generate project name
USER_ALIAS=$(az account show --query user.name -o tsv | cut -d'@' -f1 | tr '.' '-')
RANDOM_SUFFIX=$(openssl rand -hex 2)
NEW_PROJECT_NAME="${USER_ALIAS}-aiproject-${RANDOM_SUFFIX}"

# Prompt for resource group
echo "Resource group for new project:"
echo "  1. Use existing resource group: $RESOURCE_GROUP"
echo "  2. Create new resource group"

# If existing resource group
NEW_RESOURCE_GROUP="$RESOURCE_GROUP"

# Create AI Services account (hub)
HUB_NAME="${NEW_PROJECT_NAME}-hub"

echo "Creating AI Services hub: $HUB_NAME in $SELECTED_REGION..."

az cognitiveservices account create \
  --name "$HUB_NAME" \
  --resource-group "$NEW_RESOURCE_GROUP" \
  --location "$SELECTED_REGION" \
  --kind "AIServices" \
  --sku "S0" \
  --yes

# Create AI Foundry project
echo "Creating AI Foundry project: $NEW_PROJECT_NAME..."

az cognitiveservices account create \
  --name "$NEW_PROJECT_NAME" \
  --resource-group "$NEW_RESOURCE_GROUP" \
  --location "$SELECTED_REGION" \
  --kind "AIProject" \
  --sku "S0" \
  --yes

echo "✓ Project created successfully"
PROJECT_NAME="$NEW_PROJECT_NAME"
RESOURCE_GROUP="$NEW_RESOURCE_GROUP"
```

---

## Phase 7: Deploy Model

**Generate unique deployment name:**

The deployment name should match the model name (e.g., "gpt-4o"), but if a deployment with that name already exists, append a numeric suffix (e.g., "gpt-4o-2", "gpt-4o-3"). This follows the same UX pattern as Azure AI Foundry portal.

Use the `generate_deployment_name` script to check existing deployments and generate a unique name:

*Bash version:*
```bash
DEPLOYMENT_NAME=$(bash scripts/generate_deployment_name.sh \
  "$ACCOUNT_NAME" \
  "$RESOURCE_GROUP" \
  "$MODEL_NAME")

echo "Generated deployment name: $DEPLOYMENT_NAME"
```

*PowerShell version:*
```powershell
$DEPLOYMENT_NAME = & .\scripts\generate_deployment_name.ps1 `
  -AccountName $ACCOUNT_NAME `
  -ResourceGroup $RESOURCE_GROUP `
  -ModelName $MODEL_NAME

Write-Host "Generated deployment name: $DEPLOYMENT_NAME"
```

**Calculate deployment capacity:**

Follow UX capacity calculation logic. For OpenAI models, use 50% of available capacity (minimum 50 TPM). For all other models (MaaS), capacity is always 1:

```bash
if [ "$MODEL_FORMAT" = "OpenAI" ]; then
  # OpenAI models: TPM-based capacity (50% of available, minimum 50)
  SELECTED_CAPACITY=$(echo "$ALL_REGIONS_JSON" | jq -r ".value[] | select(.location==\"$SELECTED_REGION\" and .properties.skuName==\"GlobalStandard\") | .properties.availableCapacity")

  if [ "$SELECTED_CAPACITY" -gt 50 ]; then
    DEPLOY_CAPACITY=$((SELECTED_CAPACITY / 2))
    if [ "$DEPLOY_CAPACITY" -lt 50 ]; then
      DEPLOY_CAPACITY=50
    fi
  else
    DEPLOY_CAPACITY=$SELECTED_CAPACITY
  fi

  echo "Deploying with capacity: $DEPLOY_CAPACITY TPM (50% of available: $SELECTED_CAPACITY TPM)"
else
  # Non-OpenAI models (MaaS): capacity is always 1
  DEPLOY_CAPACITY=1
  echo "MaaS model — deploying with capacity: 1 (pay-per-token billing)"
fi
```

### If MODEL_FORMAT is NOT "Anthropic" — Standard CLI Deployment

> 💡 **Note:** The Azure CLI supports all non-Anthropic model formats directly.

*Bash version:*
```bash
echo "Creating deployment..."

az cognitiveservices account deployment create \
  --name "$ACCOUNT_NAME" \
  --resource-group "$RESOURCE_GROUP" \
  --deployment-name "$DEPLOYMENT_NAME" \
  --model-name "$MODEL_NAME" \
  --model-version "$MODEL_VERSION" \
  --model-format "$MODEL_FORMAT" \
  --sku-name "GlobalStandard" \
  --sku-capacity "$DEPLOY_CAPACITY"
```

*PowerShell version:*
```powershell
Write-Host "Creating deployment..."

az cognitiveservices account deployment create `
  --name $ACCOUNT_NAME `
  --resource-group $RESOURCE_GROUP `
  --deployment-name $DEPLOYMENT_NAME `
  --model-name $MODEL_NAME `
  --model-version $MODEL_VERSION `
  --model-format $MODEL_FORMAT `
  --sku-name "GlobalStandard" `
  --sku-capacity $DEPLOY_CAPACITY
```

> 💡 **Note:** For non-OpenAI MaaS models (Meta-Llama, Mistral, Cohere, etc.), `$DEPLOY_CAPACITY` is `1` (set in capacity calculation above).

### If MODEL_FORMAT is "Anthropic" — REST API Deployment with modelProviderData

The Azure CLI does not support `--model-provider-data`. You must use the ARM REST API directly.

**Step 1: Prompt user to select industry**

Present the following list and ask the user to choose one:

```
 1. None                    (API value: none)
 2. Biotechnology           (API value: biotechnology)
 3. Consulting              (API value: consulting)
 4. Education               (API value: education)
 5. Finance                 (API value: finance)
 6. Food & Beverage         (API value: food_and_beverage)
 7. Government              (API value: government)
 8. Healthcare              (API value: healthcare)
 9. Insurance               (API value: insurance)
10. Law                     (API value: law)
11. Manufacturing           (API value: manufacturing)
12. Media                   (API value: media)
13. Nonprofit               (API value: nonprofit)
14. Technology              (API value: technology)
15. Telecommunications      (API value: telecommunications)
16. Sport & Recreation      (API value: sport_and_recreation)
17. Real Estate             (API value: real_estate)
18. Retail                  (API value: retail)
19. Other                   (API value: other)
```

> ⚠️ **Do NOT pick a default industry or hardcode a value. Always ask the user.** This is required by Anthropic's terms of service. The industry list is static — there is no REST API that provides it.

Store selection as `SELECTED_INDUSTRY` (use the API value, e.g., `technology`).

**Step 2: Fetch tenant info (country code and organization name)**

```bash
TENANT_INFO=$(az rest --method GET \
  --url "https://management.azure.com/tenants?api-version=2024-11-01" \
  --query "value[0].{countryCode:countryCode, displayName:displayName}" -o json)

COUNTRY_CODE=$(echo "$TENANT_INFO" | jq -r '.countryCode')
ORG_NAME=$(echo "$TENANT_INFO" | jq -r '.displayName')
```

*PowerShell version:*
```powershell
$tenantInfo = az rest --method GET `
  --url "https://management.azure.com/tenants?api-version=2024-11-01" `
  --query "value[0].{countryCode:countryCode, displayName:displayName}" -o json | ConvertFrom-Json

$countryCode = $tenantInfo.countryCode
$orgName = $tenantInfo.displayName
```

**Step 3: Deploy via ARM REST API**

*Bash version:*
```bash
echo "Creating Anthropic model deployment via REST API..."

az rest --method PUT \
  --url "https://management.azure.com/subscriptions/$SUBSCRIPTION_ID/resourceGroups/$RESOURCE_GROUP/providers/Microsoft.CognitiveServices/accounts/$ACCOUNT_NAME/deployments/$DEPLOYMENT_NAME?api-version=2024-10-01" \
  --body "{
    \"sku\": {
      \"name\": \"GlobalStandard\",
      \"capacity\": 1
    },
    \"properties\": {
      \"model\": {
        \"format\": \"Anthropic\",
        \"name\": \"$MODEL_NAME\",
        \"version\": \"$MODEL_VERSION\"
      },
      \"modelProviderData\": {
        \"industry\": \"$SELECTED_INDUSTRY\",
        \"countryCode\": \"$COUNTRY_CODE\",
        \"organizationName\": \"$ORG_NAME\"
      }
    }
  }"
```

*PowerShell version:*
```powershell
Write-Host "Creating Anthropic model deployment via REST API..."

$body = @{
    sku = @{
        name = "GlobalStandard"
        capacity = 1
    }
    properties = @{
        model = @{
            format = "Anthropic"
            name = $MODEL_NAME
            version = $MODEL_VERSION
        }
        modelProviderData = @{
            industry = $SELECTED_INDUSTRY
            countryCode = $countryCode
            organizationName = $orgName
        }
    }
} | ConvertTo-Json -Depth 5

az rest --method PUT `
  --url "https://management.azure.com/subscriptions/$SUBSCRIPTION_ID/resourceGroups/$RESOURCE_GROUP/providers/Microsoft.CognitiveServices/accounts/$ACCOUNT_NAME/deployments/${DEPLOYMENT_NAME}?api-version=2024-10-01" `
  --body $body
```

> 💡 **Note:** Anthropic models use `capacity: 1` (MaaS billing model), not TPM-based capacity.

**Monitor deployment progress:**
```bash
echo "Monitoring deployment status..."

MAX_WAIT=300  # 5 minutes
ELAPSED=0
INTERVAL=10

while [ $ELAPSED -lt $MAX_WAIT ]; do
  STATUS=$(az cognitiveservices account deployment show \
    --name "$ACCOUNT_NAME" \
    --resource-group "$RESOURCE_GROUP" \
    --deployment-name "$DEPLOYMENT_NAME" \
    --query "properties.provisioningState" -o tsv 2>/dev/null)

  case "$STATUS" in
    "Succeeded")
      echo "✓ Deployment successful!"
      break
      ;;
    "Failed")
      echo "❌ Deployment failed"
      # Get error details
      az cognitiveservices account deployment show \
        --name "$ACCOUNT_NAME" \
        --resource-group "$RESOURCE_GROUP" \
        --deployment-name "$DEPLOYMENT_NAME" \
        --query "properties"
      exit 1
      ;;
    "Creating"|"Accepted"|"Running")
      echo "Status: $STATUS... (${ELAPSED}s elapsed)"
      sleep $INTERVAL
      ELAPSED=$((ELAPSED + INTERVAL))
      ;;
    *)
      echo "Unknown status: $STATUS"
      sleep $INTERVAL
      ELAPSED=$((ELAPSED + INTERVAL))
      ;;
  esac
done

if [ $ELAPSED -ge $MAX_WAIT ]; then
  echo "⚠ Deployment timeout after ${MAX_WAIT}s"
  echo "Check status manually:"
  echo "  az cognitiveservices account deployment show \\"
  echo "    --name $ACCOUNT_NAME \\"
  echo "    --resource-group $RESOURCE_GROUP \\"
  echo "    --deployment-name $DEPLOYMENT_NAME"
  exit 1
fi
```

---

## Phase 8: Display Deployment Details

**Show deployment information:**
```bash
echo ""
echo "═══════════════════════════════════════════"
echo "✓ Deployment Successful!"
echo "═══════════════════════════════════════════"
echo ""

# Get endpoint information
ENDPOINT=$(az cognitiveservices account show \
  --name "$ACCOUNT_NAME" \
  --resource-group "$RESOURCE_GROUP" \
  --query "properties.endpoint" -o tsv)

# Get deployment details
DEPLOYMENT_INFO=$(az cognitiveservices account deployment show \
  --name "$ACCOUNT_NAME" \
  --resource-group "$RESOURCE_GROUP" \
  --deployment-name "$DEPLOYMENT_NAME" \
  --query "properties.model")

echo "Deployment Name: $DEPLOYMENT_NAME"
echo "Model: $MODEL_NAME"
echo "Version: $MODEL_VERSION"
echo "Region: $SELECTED_REGION"
echo "SKU: GlobalStandard"
echo "Capacity: $(format_capacity $DEPLOY_CAPACITY)"
echo "Endpoint: $ENDPOINT"
echo ""

# Generate direct link to deployment in Azure AI Foundry portal
DEPLOYMENT_URL=$(bash "$(dirname "$0")/scripts/generate_deployment_url.sh" \
  --subscription "$SUBSCRIPTION_ID" \
  --resource-group "$RESOURCE_GROUP" \
  --foundry-resource "$ACCOUNT_NAME" \
  --project "$PROJECT_NAME" \
  --deployment "$DEPLOYMENT_NAME")

echo "🔗 View in Azure AI Foundry Portal:"
echo ""
echo "$DEPLOYMENT_URL"
echo ""
echo "═══════════════════════════════════════════"
echo ""

echo "Test your deployment:"
echo ""
echo "# View deployment details"
echo "az cognitiveservices account deployment show \\"
echo "  --name $ACCOUNT_NAME \\"
echo "  --resource-group $RESOURCE_GROUP \\"
echo "  --deployment-name $DEPLOYMENT_NAME"
echo ""
echo "# List all deployments"
echo "az cognitiveservices account deployment list \\"
echo "  --name $ACCOUNT_NAME \\"
echo "  --resource-group $RESOURCE_GROUP \\"
echo "  --output table"
echo ""

echo "Next steps:"
echo "• Click the link above to test in Azure AI Foundry playground"
echo "• Integrate into your application"
echo "• Set up monitoring and alerts"
```

# Preset Deployment Workflow — Step-by-Step

Condensed implementation reference for preset (optimal region) model deployment. See [SKILL.md](../SKILL.md) for overview.

**Table of Contents:** [Phase 1: Verify Authentication](#phase-1-verify-authentication) · [Phase 2: Get Current Project](#phase-2-get-current-project) · [Phase 3: Get Model Name](#phase-3-get-model-name) · [Phase 4: Check Current Region Capacity](#phase-4-check-current-region-capacity) · [Phase 5: Query Multi-Region Capacity](#phase-5-query-multi-region-capacity) · [Phase 6: Select Region and Project](#phase-6-select-region-and-project) · [Phase 7: Deploy Model](#phase-7-deploy-model)

---

## Phase 1: Verify Authentication

```bash
az account show --query "{Subscription:name, User:user.name}" -o table
```

If not logged in: `az login`

Switch subscription:

```bash
az account list --query "[].[name,id,state]" -o table
az account set --subscription <subscription-id>
```

---

## Phase 2: Get Current Project

Read `PROJECT_RESOURCE_ID` from env or prompt user. Format:
`/subscriptions/{sub-id}/resourceGroups/{rg}/providers/Microsoft.CognitiveServices/accounts/{account}/projects/{project}`

Parse ARM ID components:

```bash
SUBSCRIPTION_ID=$(echo "$PROJECT_RESOURCE_ID" | sed -n 's|.*/subscriptions/\([^/]*\).*|\1|p')
RESOURCE_GROUP=$(echo "$PROJECT_RESOURCE_ID" | sed -n 's|.*/resourceGroups/\([^/]*\).*|\1|p')
ACCOUNT_NAME=$(echo "$PROJECT_RESOURCE_ID" | sed -n 's|.*/accounts/\([^/]*\)/projects.*|\1|p')
PROJECT_NAME=$(echo "$PROJECT_RESOURCE_ID" | sed -n 's|.*/projects/\([^/?]*\).*|\1|p')
```

Verify project exists and get region:

```bash
az account set --subscription "$SUBSCRIPTION_ID"

PROJECT_REGION=$(az cognitiveservices account show \
  --name "$PROJECT_NAME" \
  --resource-group "$RESOURCE_GROUP" \
  --query location -o tsv)
```

---

## Phase 3: Get Model Name

If model not provided as parameter, list available models:

```bash
az cognitiveservices account list-models \
  --name "$PROJECT_NAME" \
  --resource-group "$RESOURCE_GROUP" \
  --query "[].name" -o tsv | sort -u
```

Get versions for selected model:

```bash
az cognitiveservices account list-models \
  --name "$PROJECT_NAME" \
  --resource-group "$RESOURCE_GROUP" \
  --query "[?name=='$MODEL_NAME'].{Name:name, Version:version, Format:format}" \
  -o table
```

---

## Phase 4: Check Current Region Capacity

```bash
CAPACITY_JSON=$(az rest --method GET \
  --url "https://management.azure.com/subscriptions/$SUBSCRIPTION_ID/providers/Microsoft.CognitiveServices/locations/$PROJECT_REGION/modelCapacities?api-version=2024-10-01&modelFormat=OpenAI&modelName=$MODEL_NAME&modelVersion=$MODEL_VERSION")

CURRENT_CAPACITY=$(echo "$CAPACITY_JSON" | jq -r '.value[] | select(.properties.skuName=="GlobalStandard") | .properties.availableCapacity')
```

If `CURRENT_CAPACITY > 0` → skip to Phase 7. Otherwise continue to Phase 5.

---

## Phase 5: Query Multi-Region Capacity

```bash
ALL_REGIONS_JSON=$(az rest --method GET \
  --url "https://management.azure.com/subscriptions/$SUBSCRIPTION_ID/providers/Microsoft.CognitiveServices/modelCapacities?api-version=2024-10-01&modelFormat=OpenAI&modelName=$MODEL_NAME&modelVersion=$MODEL_VERSION")
```

Extract available regions (capacity > 0):

```bash
AVAILABLE_REGIONS=$(echo "$ALL_REGIONS_JSON" | jq -r '.value[] | select(.properties.skuName=="GlobalStandard" and .properties.availableCapacity > 0) | "\(.location)|\(.properties.availableCapacity)"')
```

Extract unavailable regions:

```bash
UNAVAILABLE_REGIONS=$(echo "$ALL_REGIONS_JSON" | jq -r '.value[] | select(.properties.skuName=="GlobalStandard" and (.properties.availableCapacity == 0 or .properties.availableCapacity == null)) | "\(.location)|0"')
```

If no regions have capacity, defer to the [quota skill](../../../../quota/quota.md) for increase requests. Suggest checking existing deployments or trying alternative models like `gpt-4o-mini`.

---

## Phase 6: Select Region and Project

Present available regions to user. Store selection as `SELECTED_REGION`.

Find projects in selected region:

```bash
PROJECTS_IN_REGION=$(az cognitiveservices account list \
  --query "[?kind=='AIProject' && location=='$SELECTED_REGION'].{Name:name, ResourceGroup:resourceGroup}" \
  --output json)
```

**If no projects exist — create new:**

```bash
az cognitiveservices account create \
  --name "$HUB_NAME" \
  --resource-group "$RESOURCE_GROUP" \
  --location "$SELECTED_REGION" \
  --kind "AIServices" \
  --sku "S0" --yes

az cognitiveservices account create \
  --name "$NEW_PROJECT_NAME" \
  --resource-group "$RESOURCE_GROUP" \
  --location "$SELECTED_REGION" \
  --kind "AIProject" \
  --sku "S0" --yes
```

---

## Phase 7: Deploy Model

Generate unique deployment name using `scripts/generate_deployment_name.sh`:

```bash
DEPLOYMENT_NAME=$(bash scripts/generate_deployment_name.sh "$ACCOUNT_NAME" "$RESOURCE_GROUP" "$MODEL_NAME")
```

Calculate capacity — 50% of available, minimum 50 TPM:

```bash
SELECTED_CAPACITY=$(echo "$ALL_REGIONS_JSON" | jq -r ".value[] | select(.location==\"$SELECTED_REGION\" and .properties.skuName==\"GlobalStandard\") | .properties.availableCapacity")
DEPLOY_CAPACITY=$(( SELECTED_CAPACITY / 2 ))
[ "$DEPLOY_CAPACITY" -lt 50 ] && DEPLOY_CAPACITY=50
```

Create deployment:

```bash
az cognitiveservices account deployment create \
  --name "$ACCOUNT_NAME" \
  --resource-group "$RESOURCE_GROUP" \
  --deployment-name "$DEPLOYMENT_NAME" \
  --model-name "$MODEL_NAME" \
  --model-version "$MODEL_VERSION" \
  --model-format "OpenAI" \
  --sku-name "GlobalStandard" \
  --sku-capacity "$DEPLOY_CAPACITY"
```

Monitor with `az cognitiveservices account deployment show ... --query "properties.provisioningState"` until `Succeeded` or `Failed`.

# Generate Azure AI Foundry portal URL for a model deployment
# This script creates a direct clickable link to view a deployment in the Azure AI Foundry portal
#
# NOTE: The encoding scheme for the subscription ID portion is proprietary to Azure AI Foundry.
# This script uses a GUID byte encoding approach, but may need adjustment based on the actual encoding used.

param(
    [Parameter(Mandatory=$true)]
    [string]$SubscriptionId,
    
    [Parameter(Mandatory=$true)]
    [string]$ResourceGroup,
    
    [Parameter(Mandatory=$true)]
    [string]$FoundryResource,
    
    [Parameter(Mandatory=$true)]
    [string]$ProjectName,
    
    [Parameter(Mandatory=$true)]
    [string]$DeploymentName
)

function Get-SubscriptionIdEncoded {
    param([string]$SubscriptionId)
    
    # Parse GUID and convert to bytes in string order (big-endian)
    # Not using ToByteArray() because it uses little-endian format
    $guidString = $SubscriptionId.Replace('-', '')
    $bytes = New-Object byte[] 16
    for ($i = 0; $i -lt 16; $i++) {
        $bytes[$i] = [Convert]::ToByte($guidString.Substring($i * 2, 2), 16)
    }
    
    # Encode as base64url
    $base64 = [Convert]::ToBase64String($bytes)
    $urlSafe = $base64.Replace('+', '-').Replace('/', '_').TrimEnd('=')
    return $urlSafe
}

function Get-FoundryDeploymentUrl {
    param(
        [string]$SubscriptionId,
        [string]$ResourceGroup,
        [string]$FoundryResource,
        [string]$ProjectName,
        [string]$DeploymentName
    )
    
    # Encode subscription ID
    $encodedSubId = Get-SubscriptionIdEncoded -SubscriptionId $SubscriptionId
    
    # Build the encoded resource path
    # Format: {encoded-sub-id},{resource-group},,{foundry-resource},{project-name}
    # Note: Two commas between resource-group and foundry-resource
    $encodedPath = "$encodedSubId,$ResourceGroup,,$FoundryResource,$ProjectName"
    
    # Build the full URL
    $baseUrl = "https://ai.azure.com/nextgen/r/"
    $deploymentPath = "/build/models/deployments/$DeploymentName/details"
    
    return "$baseUrl$encodedPath$deploymentPath"
}

# Generate and output the URL
$url = Get-FoundryDeploymentUrl `
    -SubscriptionId $SubscriptionId `
    -ResourceGroup $ResourceGroup `
    -FoundryResource $FoundryResource `
    -ProjectName $ProjectName `
    -DeploymentName $DeploymentName

Write-Output $url

#!/bin/bash
# Generate Azure AI Foundry portal URL for a model deployment
# This script creates a direct clickable link to view a deployment in the Azure AI Foundry portal

set -e

# Function to display usage
usage() {
    cat << EOF
Usage: $0 --subscription SUBSCRIPTION_ID --resource-group RESOURCE_GROUP \\
          --foundry-resource FOUNDRY_RESOURCE --project PROJECT_NAME \\
          --deployment DEPLOYMENT_NAME

Generate Azure AI Foundry deployment URL

Required arguments:
  --subscription        Azure subscription ID (GUID)
  --resource-group      Resource group name
  --foundry-resource    Foundry resource (account) name
  --project             Project name
  --deployment          Deployment name

Example:
  $0 --subscription d5320f9a-73da-4a74-b639-83efebc7bb6f \\
     --resource-group bani-host \\
     --foundry-resource banide-host-resource \\
     --project banide-host \\
     --deployment text-embedding-ada-002
EOF
    exit 1
}

# Parse command line arguments
while [[ $# -gt 0 ]]; do
    case $1 in
        --subscription)
            SUBSCRIPTION_ID="$2"
            shift 2
            ;;
        --resource-group)
            RESOURCE_GROUP="$2"
            shift 2
            ;;
        --foundry-resource)
            FOUNDRY_RESOURCE="$2"
            shift 2
            ;;
        --project)
            PROJECT_NAME="$2"
            shift 2
            ;;
        --deployment)
            DEPLOYMENT_NAME="$2"
            shift 2
            ;;
        -h|--help)
            usage
            ;;
        *)
            echo "Unknown option: $1"
            usage
            ;;
    esac
done

# Validate required arguments
if [ -z "$SUBSCRIPTION_ID" ] || [ -z "$RESOURCE_GROUP" ] || [ -z "$FOUNDRY_RESOURCE" ] || \
   [ -z "$PROJECT_NAME" ] || [ -z "$DEPLOYMENT_NAME" ]; then
    echo "Error: Missing required arguments"
    usage
fi

# Convert subscription GUID to bytes (big-endian/string order) and encode as base64url
# Remove hyphens from GUID
GUID_HEX=$(echo "$SUBSCRIPTION_ID" | tr -d '-')

# Convert hex string to bytes and base64 encode
# Using xxd to convert hex to binary, then base64 encode
ENCODED_SUB=$(echo "$GUID_HEX" | xxd -r -p | base64 | tr '+' '-' | tr '/' '_' | tr -d '=')

# Build the encoded resource path
# Format: {encoded-sub-id},{resource-group},,{foundry-resource},{project-name}
# Note: Two commas between resource-group and foundry-resource
ENCODED_PATH="${ENCODED_SUB},${RESOURCE_GROUP},,${FOUNDRY_RESOURCE},${PROJECT_NAME}"

# Build the full URL
BASE_URL="https://ai.azure.com/nextgen/r/"
DEPLOYMENT_PATH="/build/models/deployments/${DEPLOYMENT_NAME}/details"

echo "${BASE_URL}${ENCODED_PATH}${DEPLOYMENT_PATH}"

# Foundry Project Connections

Connections authenticate and link external resources to a Foundry project. Many agent tools (Azure AI Search, Bing Grounding, MCP) require a project connection before use.

## Managing Connections via MCP

Use the Foundry MCP server for all connection operations. The MCP tools handle authentication, validation, and project scoping automatically.

| Operation | MCP Tool | Description |
|-----------|----------|-------------|
| List all connections | `project_connection_list` | Lists project connections and can filter by category or target |
| Get connection details | `project_connection_get` | Retrieves a specific connection by `connectionName` |
| Create a connection | `project_connection_create` | Creates or replaces a project connection to an external resource |
| Update a connection | `project_connection_update` | Updates auth, category, target, or expiry on an existing connection |
| Delete a connection | `project_connection_delete` | Removes a connection from the project by name |
| List supported categories/auth types | `project_connection_list_metadata` | Lists valid connection categories and auth types before create/update |

> 💡 **Tip:** Use `project_connection_get` or `project_connection_list` to resolve the connection name and full connection resource ID before configuring agent tools that require `project_connection_id`.

## Create Connection via Portal

1. Open [Microsoft Foundry portal](https://ai.azure.com)
2. Navigate to **Operate** → **Admin** → select your project
3. Select **Add connection** → choose service type
4. Browse for resource, select auth method, click **Add connection**

## Connection ID Format

For REST and TypeScript samples, the full connection ID format is:

```
/subscriptions/{subId}/resourceGroups/{rg}/providers/Microsoft.CognitiveServices/accounts/{account}/projects/{project}/connections/{connectionName}
```

Python and C# SDKs resolve this automatically from the connection name.

## Common Connection Types

| Type | Resource | Used By |
|------|----------|---------|
| `azure_ai_search` | Azure AI Search | AI Search tool |
| `bing` | Grounding with Bing Search | Bing grounding tool |
| `bing_custom_search` | Grounding with Bing Custom Search | Bing Custom Search tool |
| `api_key` | Any API-key resource | MCP servers, custom tools |
| `azure_openai` | Azure OpenAI | Model access |
| `AzureStorageAccount` | Azure Blob Storage | Dataset upload via `evaluation_dataset_create` |

## RBAC for Connection Management

| Role | Scope | Permission |
|------|-------|------------|
| **Foundry Project Manager** | Project | Create/manage project connections |
| **Contributor** or **Owner** | Subscription/RG | Create Bing/Search resources, get keys |

## Troubleshooting

| Error | Cause | Fix |
|-------|-------|-----|
| `Connection not found` | Name mismatch or wrong project | Use `project_connection_list` to find the correct `connectionName` |
| `Unauthorized` creating connection | Missing Foundry Project Manager role | Assign role on the Foundry project |
| `Invalid connection ID format` | Using name instead of full resource ID | Use `project_connection_get` to resolve the full ID |

---
name: foundry-create-project
description: |
  Create a new Azure AI Foundry project using Azure Developer CLI (azd) to provision infrastructure for hosting AI agents and models.
  USE FOR: create Foundry project, new AI Foundry project, set up Foundry, azd init Foundry, provision Foundry infrastructure, onboard to Foundry, create Azure AI project, set up AI project.
  DO NOT USE FOR: deploying agents to existing projects (use agent/deploy), creating agent code (use agent/create), deploying AI models from catalog (use microsoft-foundry main skill), Azure Functions (use azure-functions).
allowed-tools: Read, Write, Bash, AskUserQuestion
---

# Create Azure AI Foundry Project

Create a new Azure AI Foundry project using azd. Provisions: Foundry account, project, Application Insights, managed identity, and RBAC permissions. Optionally enables hosted-agent deployment (adds an Azure Container Registry, and — only when the **Standard Setup** capability-host flag is also enabled — a `capabilityHosts/agents` resource).

**Table of Contents:** [Prerequisites](#prerequisites) · [Workflow](#workflow) · [Best Practices](#best-practices) · [Troubleshooting](#troubleshooting) · [Related Skills](#related-skills) · [Resources](#resources)

## Prerequisites

Run checks in order. STOP on any failure and resolve before proceeding.

**1. Azure CLI** — `az version` → expects version output. If missing: https://aka.ms/installazurecli

**2. Azure login & subscription:**

```bash
az account show --query "{Name:name, SubscriptionId:id, State:state}" -o table
```

If not logged in, run `az login`. If no active subscription: https://azure.microsoft.com/free/ — STOP.

If multiple subscriptions, ask which to use, then `az account set --subscription "<id>"`.

**3. Role permissions:**

```bash
az role assignment list --assignee "$(az ad signed-in-user show --query id -o tsv)" --query "[?contains(roleDefinitionName, 'Owner') || contains(roleDefinitionName, 'Contributor') || contains(roleDefinitionName, 'Foundry')].{Role:roleDefinitionName, Scope:scope}" -o table
```

Requires Owner, Contributor, or Foundry Owner. If insufficient — STOP, request elevated access from admin.

**4. Azure Developer CLI** — `azd version`. If missing: https://aka.ms/azure-dev/install

## Workflow

### Step 1: Verify azd login

```bash
azd auth login --check-status
```

If not logged in, run `azd auth login` and complete browser auth.

### Step 2: Resolve Project Details

Collect only values the user has not already provided. For values not specified, use defaults:

1. **Project name** — used as azd environment name and resource group (`rg-<name>`). Must contain only alphanumeric characters and hyphens.
   - If the user provided a name, use it as-is.
   - If the user did NOT provide a name, **auto-generate a unique name** using the pattern `ai-project-<random>` where `<random>` is a short random suffix (6-8 lowercase alphanumeric characters). Generate the suffix with a platform-appropriate method:
     ```bash
     # bash/zsh
     echo "ai-project-$(openssl rand -hex 4)"
     ```
     ```powershell
     # PowerShell
     "ai-project-$(-join ((48..57)+(97..122) | Get-Random -Count 8 | ForEach-Object {[char]$_}))"
     ```
   - Show the generated name to the user before proceeding, but do not block on confirmation — proceed unless the user objects.
   - Examples: `ai-project-3f8a1b2c`, `my-ai-project`, `dev-agents`
2. **Azure location** (optional) — defaults to North Central US (required for hosted agents preview)
3. **Enable hosted agents?** (yes/no) — enables hosted-agent deployment and provisions an Azure Container Registry. A capability host (`capabilityHosts/agents`, used by Foundry's **Standard Agent Setup** for bring-your-own storage) is also created only when `ENABLE_CAPABILITY_HOST=true`. Defaults to no. See [Step 3](#step-3-create-directory-and-initialize) for how the two flags interact.

### Step 3: Create Directory and Initialize

```bash
mkdir "<project-name>" && cd "<project-name>"
azd init -t https://github.com/Azure-Samples/azd-ai-starter-basic -e <project-name> --no-prompt
```

- `-t` — Azure AI starter template (Foundry infrastructure)
- `-e` — environment name
- `--no-prompt` — non-interactive, use defaults
- **IMPORTANT:** `azd init` requires an empty directory

If user specified a non-default location:

```bash
azd config set defaults.location <location>
```

If user chose to enable hosted agents:

```bash
azd env set ENABLE_HOSTED_AGENTS true
azd env set ENABLE_CAPABILITY_HOST false
```

`ENABLE_HOSTED_AGENTS=true` enables hosted-agent deployment and creates an Azure Container Registry for the container image. A capability host (`capabilityHosts/agents`, used by Foundry's **Standard Agent Setup** for bring-your-own storage) is **also** created only when `ENABLE_CAPABILITY_HOST=true`. The default `azd ai agent` flow targets **Basic Agent Setup**, so it sets `ENABLE_CAPABILITY_HOST=false` automatically. The two flags are independent.

> ⚠️ **Warning:** The Bicep template parameter `enableCapabilityHost` defaults to `true`. If you set `ENABLE_HOSTED_AGENTS` by hand without also setting `ENABLE_CAPABILITY_HOST=false`, you will accidentally provision Standard Setup (with the capability host). Use `azd ai agent init` to set both flags correctly.

See the canonical env-var docs: [azure-dev/cli/azd/docs/environment-variables.md](https://github.com/Azure/azure-dev/blob/main/cli/azd/docs/environment-variables.md).

### Step 4: Provision Infrastructure

```bash
azd provision --no-prompt
```

Takes 5–10 minutes. Creates resource group, Foundry account/project, Application Insights, managed identity, and RBAC roles. If `ENABLE_HOSTED_AGENTS=true`, also creates an Azure Container Registry. A `capabilityHosts/agents` resource is created **only** when `ENABLE_CAPABILITY_HOST=true` (Standard Setup); the default Basic Setup uses `ENABLE_CAPABILITY_HOST=false` and no capability host is provisioned — its absence is correct.

### Step 5: Retrieve Project Details

```bash
azd env get-values
```

Capture `AZURE_AI_PROJECT_ID`, `AZURE_AI_PROJECT_ENDPOINT`, and `AZURE_RESOURCE_GROUP`. Direct user to verify at https://ai.azure.com.

### Step 6: Next Steps

> **Next — azd Golden Path:** create a hosted agent with [foundry-agent/create/create-hosted.md](../../foundry-agent/create/create-hosted.md). For headless / scripted flows, **pre-bootstrap the workspace with core `azd init`** so subscription + location are populated before model resolution runs:
>
> ```bash
> azd init -t Azure-Samples/azd-ai-starter-basic . -e <env-name> --subscription <id> -l <region>
> azd ai agent init -m <manifest-url> --no-prompt --deploy-mode code --runtime python_3_13 --entry-point main.py
> ```
>
> Core `azd init` accepts `--subscription` and `-l/--location`; `azd ai agent init` does not. `azd ai agent init` then resolves the model from the chosen sample's manifest and writes it into `azure.yaml services.ai-project.deployments[]`; the next `azd provision` creates the deployment through Bicep. **You do not need to deploy a model separately for this path** — no `az cognitiveservices` calls, no `azd env set AI_PROJECT_DEPLOYMENTS`.
>
> Use [models/deploy-model](../../models/deploy-model/SKILL.md) **only** for out-of-band scenarios: adding models to a Foundry project that is not managed by this azd project, or ad-hoc deployments outside the azd lifecycle.

- Deploy an existing agent → [foundry-agent/deploy/deploy.md](../../foundry-agent/deploy/deploy.md)
- Browse model catalog → `foundry_models_list` MCP tool
- Manage project → https://ai.azure.com

## Best Practices

- Use North Central US for hosted agents (preview requirement)
- Name must be alphanumeric + hyphens only — no spaces, underscores, or special characters
- Delete unused projects with `azd down` to avoid ongoing costs
- `azd down` deletes ALL resources — Foundry account, agents, models, Container Registry, and Application Insights data
- `azd provision` is safe to re-run on failure

## Troubleshooting

| Problem | Solution |
|---------|----------|
| `azd: command not found` | Install from https://aka.ms/azure-dev/install |
| `ERROR: Failed to authenticate` | Run `azd auth login`; verify subscription with `az account list` |
| `environment name '' is invalid` | Name must be alphanumeric + hyphens only |
| `ERROR: Insufficient permissions` | Request Contributor or Foundry Owner role from admin |
| Region not supported for hosted agents | Use `azd config set defaults.location northcentralus` |
| Provisioning timeout | Check region availability, verify connectivity, retry `azd provision` |

## Related Skills

- **agent/deploy** — Deploy agents to the created project
- **agent/create** — Create a new agent for deployment

## Resources

- [Azure Developer CLI](https://aka.ms/azure-dev/install) · [AI Foundry Portal](https://ai.azure.com) · [Foundry Docs](https://learn.microsoft.com/azure/ai-foundry/) · [azd-ai-starter-basic template](https://github.com/Azure-Samples/azd-ai-starter-basic)

# Microsoft Foundry Quota Management

Quota and capacity management for Microsoft Foundry. Quotas are **subscription + region** level.

> ⚠️ **Important:** This is the **authoritative skill** for all Foundry quota operations. When a user asks about quota, capacity, TPM, PTU, quota errors, or deployment limits, **always invoke this skill** rather than using MCP tools (azure-quota, azure-documentation, azure-foundry) directly. This skill provides structured workflows and error handling that direct tool calls lack.

> **Important:** All quota operations are **control plane (management)** operations. Use **Azure CLI commands** (`az cognitiveservices`, `az rest`, `az ai`) as the primary method.

## Quota Types

| Type | Description |
|------|-------------|
| **TPM** | Tokens Per Minute, pay-per-token, subject to rate limits |
| **PTU** | Provisioned Throughput Units, monthly commitment, no rate limits |
| **Region** | Max capacity per region, shared across subscription |
| **Slots** | 10-20 deployment slots per resource |

**When to use PTU:** Consistent high-volume production workloads where monthly commitment is cost-effective.

---

Use this sub-skill when the user needs to:

- **View quota usage** — check current TPM/PTU allocation and available capacity
- **Check quota limits** — show quota limits for a subscription, region, or model
- **Find optimal regions** — compare quota availability across regions for deployment
- **Plan deployments** — verify sufficient quota before deploying models
- **Request quota increases** — navigate quota increase process through Azure Portal
- **Troubleshoot deployment failures** — diagnose QuotaExceeded, InsufficientQuota, DeploymentLimitReached, 429 rate limit errors
- **Optimize allocation** — monitor and consolidate quota across deployments
- **Monitor quota across deployments** — track capacity by model and region
- **Explain quota concepts** — explain TPM, PTU, capacity units, regional quotas
- **Free up quota** — identify and delete unused deployments

**Key Points:**
1. Isolated by region (East US ≠ West US)
2. Regional capacity varies by model
3. Multi-region enables failover and load distribution
4. Quota requests specify target region

See [detailed guide](./references/workflows.md#regional-quota).

---

## Core Workflows

### 1. Check Regional Quota

```bash
subId=$(az account show --query id -o tsv)
az rest --method get \
  --url "https://management.azure.com/subscriptions/$subId/providers/Microsoft.CognitiveServices/locations/eastus/usages?api-version=2023-05-01" \
  --query "value[?contains(name.value,'OpenAI')].{Model:name.value, Used:currentValue, Limit:limit}" -o table
```

**Output interpretation:**
- **Used**: Current TPM consumed (10000 = 10K TPM)
- **Limit**: Maximum TPM quota (15000 = 15K TPM)
- **Available**: Limit - Used (5K TPM available)

Change region: `eastus`, `eastus2`, `westus`, `westus2`, `swedencentral`, `uksouth`.

---

### 2. Find Best Region for Deployment

Check specific regions for available quota:

```bash
subId=$(az account show --query id -o tsv)
region="eastus"
az rest --method get \
  --url "https://management.azure.com/subscriptions/$subId/providers/Microsoft.CognitiveServices/locations/$region/usages?api-version=2023-05-01" \
  --query "value[?name.value=='OpenAI.Standard.gpt-4o'].{Model:name.value, Used:currentValue, Limit:limit, Available:(limit-currentValue)}" -o table
```

See [workflows reference](./references/workflows.md#multi-region-check) for multi-region comparison.

---

### 3. Check Quota Before Deployment

Verify available quota for your target model:

```bash
subId=$(az account show --query id -o tsv)
region="eastus"
model="OpenAI.Standard.gpt-4o"

az rest --method get \
  --url "https://management.azure.com/subscriptions/$subId/providers/Microsoft.CognitiveServices/locations/$region/usages?api-version=2023-05-01" \
  --query "value[?name.value=='$model'].{Model:name.value, Used:currentValue, Limit:limit, Available:(limit-currentValue)}" -o table
```

- **Available > 0**: Yes, you have quota
- **Available = 0**: Delete unused deployments or try different region

---

### 4. Monitor Quota by Model

Show quota allocation grouped by model:

```bash
subId=$(az account show --query id -o tsv)
region="eastus"
az rest --method get \
  --url "https://management.azure.com/subscriptions/$subId/providers/Microsoft.CognitiveServices/locations/$region/usages?api-version=2023-05-01" \
  --query "value[?contains(name.value,'OpenAI')].{Model:name.value, Used:currentValue, Limit:limit, Available:(limit-currentValue)}" -o table
```

Shows aggregate usage across ALL deployments by model type.

**Optional:** List individual deployments:
- **Azure MCP tool**: Use `model_deployment_get` to query deployments in a Foundry project
- **Azure CLI**:
```bash
az cognitiveservices account list --query "[?kind=='AIServices'].{Name:name,RG:resourceGroup}" -o table

az cognitiveservices account deployment list --name <resource> --resource-group <rg> \
  --query "[].{Name:name,Model:properties.model.name,Capacity:sku.capacity}" -o table
```

---

### 5. Delete Deployment (Free Quota)

```bash
az cognitiveservices account deployment delete --name <resource> --resource-group <rg> \
  --deployment-name <deployment>
```

Quota freed **immediately**. Re-run Workflow #1 to verify.

---

### 6. Request Quota Increase

**Azure Portal Process:**
1. Navigate to [Azure Portal - All Resources](https://portal.azure.com/#view/HubsExtension/BrowseAll) → Filter "AI Services" → Click resource
2. Select **Quotas** in left navigation
3. Click **Request quota increase**
4. Fill form: Model, Current Limit, Requested Limit, Region, **Business Justification** (required field)
5. Wait for approval: **3-5 business days typically, up to 10 business days** ([source](https://learn.microsoft.com/en-us/azure/ai-foundry/openai/how-to/quota))

**Business Justification** is a mandatory field that explains why you need more quota. Azure reviews each request to ensure resources are allocated based on legitimate business needs. A strong justification includes:
- **Workload details**: What you're building and which model you need
- **Data-driven estimates**: Expected traffic volume and token usage calculations
- **Clear need**: Why current quota is insufficient and what capacity you require
- **Timeline**: When you need the increased quota (e.g., production launch date)

**Business Justification template:**
```
Production [workload type] using [model] in [region].
Expected traffic: [X requests/day] with [Y tokens/request].
Calculated required TPM: [Z TPM]. Current [N TPM] insufficient.
Request increase to [M TPM]. Deployment target: [date].
```

See [detailed quota request guide](./references/workflows.md#request-quota-increase) for complete steps.

---

## Quick Troubleshooting

| Error | Quick Fix | Detailed Guide |
|-------|-----------|----------------|
| `QuotaExceeded` | Delete unused deployments or request increase | [Error Resolution](./references/error-resolution.md#quotaexceeded) |
| `InsufficientQuota` | Reduce capacity or try different region | [Error Resolution](./references/error-resolution.md#insufficientquota) |
| `DeploymentLimitReached` | Delete unused deployments (10-20 slot limit) | [Error Resolution](./references/error-resolution.md#deploymentlimitreached) |
| `429 Rate Limit` | Increase TPM or migrate to PTU | [Error Resolution](./references/error-resolution.md#429-errors) |

---

## References

**Detailed Guides:**
- [Error Resolution Workflows](./references/error-resolution.md) - Detailed workflows for quota exhausted, 429 errors, insufficient quota, deployment limits
- [Troubleshooting Guide](./references/troubleshooting.md) - Quick error fixes and diagnostic commands
- [Quota Optimization Strategies](./references/optimization.md) - 5 strategies for freeing quota and reducing costs
- [Capacity Planning Guide](./references/capacity-planning.md) - TPM vs PTU comparison, model selection, workload calculations
- [Workflows Reference](./references/workflows.md) - Complete workflow steps and multi-region checks
- [PTU Guide](./references/ptu-guide.md) - Provisioned throughput capacity planning

**Official Microsoft Documentation:**
- [Azure OpenAI Service Pricing](https://azure.microsoft.com/en-us/pricing/details/cognitive-services/openai-service/) - Official pay-per-token rates
- [PTU Costs and Billing](https://learn.microsoft.com/en-us/azure/ai-foundry/openai/how-to/provisioned-throughput-onboarding) - PTU hourly rates
- [Azure OpenAI Models](https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/models) - Model capabilities and regions
- [Quota Management Guide](https://learn.microsoft.com/azure/ai-services/openai/how-to/quota) - Official quota procedures
- [Quotas and Limits](https://learn.microsoft.com/azure/ai-services/openai/quotas-limits) - Rate limits and quota details

**Calculators:**
- [Azure Pricing Calculator](https://azure.microsoft.com/pricing/calculator/) - Official pricing estimator
- Azure AI Foundry PTU calculator (Microsoft Foundry → Operate → Quota → Provisioned Throughput Unit tab) - PTU capacity sizing

# Capacity Planning Guide

Comprehensive guide for planning Azure AI Foundry capacity, including cost analysis, model selection, and workload calculations.

**Table of Contents:** [Cost Comparison: TPM vs PTU](#cost-comparison-tpm-vs-ptu) · [Production Workload Examples](#production-workload-examples) · [Model Selection and Deployment Type Guidance](#model-selection-and-deployment-type-guidance)

## Cost Comparison: TPM vs PTU

> **Official Pricing Sources:**
> - [Azure OpenAI Service Pricing](https://azure.microsoft.com/en-us/pricing/details/cognitive-services/openai-service/) - Official pay-per-token rates
> - [PTU Costs and Billing Guide](https://learn.microsoft.com/en-us/azure/ai-foundry/openai/how-to/provisioned-throughput-onboarding) - PTU hourly rates and capacity planning

**TPM (Standard) Pricing:**
- Pay-per-token for input/output
- No upfront commitment
- **Rates**: See [Azure OpenAI Pricing](https://azure.microsoft.com/en-us/pricing/details/cognitive-services/openai-service/)
  - GPT-4o: ~$0.0025-$0.01/1K tokens
  - GPT-4 Turbo: ~$0.01-$0.03/1K
  - GPT-3.5 Turbo: ~$0.0005-$0.0015/1K
- **Best for**: Variable workloads, unpredictable traffic

**PTU (Provisioned) Pricing:**
- Hourly billing: `$/PTU/hr × PTUs × 730 hrs/month`
- Monthly commitment with Reservations discounts
- **Rates**: See [PTU Billing Guide](https://learn.microsoft.com/en-us/azure/ai-foundry/openai/how-to/provisioned-throughput-onboarding)
- Use PTU calculator to determine requirements (Microsoft Foundry → Operate → Quota → Provisioned Throughput Unit tab)
- **Best for**: High-volume (>1M tokens/day), predictable traffic, guaranteed throughput

**Cost Decision Framework** (Analytical Guidance):

```
Step 1: Calculate monthly TPM cost
  Monthly TPM cost = (Daily tokens × 30 days × $price per 1K tokens) / 1000

Step 2: Calculate monthly PTU cost
  Monthly PTU cost = Required PTUs × 730 hours/month × $PTU-hour rate
  (Get Required PTUs from Azure AI Foundry portal: Microsoft Foundry → Operate → Quota → Provisioned Throughput Unit tab)

Step 3: Compare
  Use PTU when: Monthly PTU cost < (Monthly TPM cost × 0.7)
  (Use 70% threshold to account for commitment risk)
```

**Example Calculation** (Analytical):

Scenario: 1M requests/day, average 1,000 tokens per request

- **Daily tokens**: 1,000,000 × 1,000 = 1B tokens/day
- **TPM Cost** (using GPT-4o at $0.005/1K avg): (1B × 30 × $0.005) / 1000 = ~$150,000/month
- **PTU Cost** (estimated 100 PTU at ~$5/PTU-hour): 100 PTU × 730 hours × $5 = ~$365,000/month
- **Decision**: Use TPM (significantly lower cost for this workload)

> **Important**: Always use the official [Azure Pricing Calculator](https://azure.microsoft.com/pricing/calculator/) and Azure AI Foundry portal PTU calculator (Microsoft Foundry → Operate → Quota → Provisioned Throughput Unit tab) for exact pricing by model, region, and workload. Prices vary by region and are subject to change.

---

## Production Workload Examples

To estimate quota requirements, use real-world production scenarios with capacity calculations for gpt-4, version 0613 (from Azure Foundry Portal calculator):

| Workload Type | Calls/Min | Prompt Tokens | Response Tokens | Cache Hit % | Total Tokens/Min | PTU Required | TPM Equivalent |
|---------------|-----------|---------------|-----------------|-------------|------------------|--------------|----------------|
| **RAG Chat** | 10 | 3,500 | 300 | 20% | 38,000 | 100 | 38K TPM |
| **Basic Chat** | 10 | 500 | 100 | 20% | 6,000 | 100 | 6K TPM |
| **Summarization** | 10 | 5,000 | 300 | 20% | 53,000 | 100 | 53K TPM |
| **Classification** | 10 | 3,800 | 10 | 20% | 38,100 | 100 | 38K TPM |

**How to Estimate Your Production Quota Requirements:**

To calculate your quota needs for production deployments, follow these steps:

1. **Determine your peak calls per minute**: Monitor or estimate maximum concurrent requests
2. **Measure token usage**: Average prompt size + response size
3. **Account for cache hits**: Prompt caching can reduce effective token count by 20-50%
4. **Calculate total tokens/min**: (Calls/min × (Prompt tokens + Response tokens)) × (1 - Cache %)
5. **Choose deployment type**:
   - **TPM (Standard)**: Allocate 1.5-2× your calculated tokens/min for headroom
   - **PTU (Provisioned)**: Use Azure AI Foundry portal PTU calculator for exact PTU count (Microsoft Foundry → Operate → Quota → Provisioned Throughput Unit tab)

**Example Calculation (RAG Chat Production):**
- Peak: 10 calls/min
- Prompt: 3,500 tokens (context + question)
- Response: 300 tokens (answer)
- Cache: 20% hit rate (reduces prompt tokens by 20%)
- **Total TPM needed**: (10 × (3,500 × 0.8 + 300)) = 31,000 TPM
- **With 50% headroom**: 46,500 TPM → Round to **50K TPM deployment**

**PTU Recommendation:**
For the combined workload (40 calls/min, 135K tokens/min total), use **200 PTU** (from calculator above).

---

## Model Selection and Deployment Type Guidance

> **Official Documentation:**
> - [Choose the Right AI Model for Your Workload](https://learn.microsoft.com/en-us/azure/architecture/ai-ml/guide/choose-ai-model) - Microsoft Architecture Center
> - [Azure OpenAI Models](https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/models) - Model capabilities, regions, and quotas
> - [Understanding Deployment Types](https://learn.microsoft.com/en-us/azure/ai-foundry/foundry-models/concepts/deployment-types) - Standard vs Provisioned guidance

**Model Characteristics** (from [official Azure OpenAI documentation](https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/models)):

| Model | Key Characteristics | Best For |
|-------|---------------------|----------|
| **GPT-4o** | Matches GPT-4 Turbo performance in English text/coding, superior in non-English and vision tasks. Cheaper and faster than GPT-4 Turbo. | Multimodal tasks, cost-effective general purpose, high-volume production workloads |
| **GPT-4 Turbo** | Superior reasoning capabilities, larger context window (128K tokens) | Complex reasoning tasks, long-context analysis |
| **GPT-3.5 Turbo** | Most cost-effective, optimized for chat and completions, fast response time | Simple tasks, customer service, high-volume low-cost scenarios |
| **GPT-4o mini** | Fastest response time, low latency | Latency-sensitive applications requiring immediate responses |
| **text-embedding-3-large** | Purpose-built for vector embeddings | RAG applications, semantic search, document similarity |

**Deployment Type Selection** (from [official deployment types guide](https://learn.microsoft.com/en-us/azure/ai-foundry/foundry-models/concepts/deployment-types)):

| Traffic Pattern | Recommended Deployment Type | Reason |
|-----------------|---------------------------|---------|
| **Variable, bursty traffic** | Standard or Global Standard (pay-per-token) | No commitment, pay only for usage |
| **Consistent high volume** | Provisioned types (PTU) | Reserved capacity, predictable costs |
| **Large batch jobs (non-time-sensitive)** | Global Batch or DataZone Batch | 50% cost savings vs Standard |
| **Low latency variance required** | Provisioned types | Guaranteed throughput, no rate limits |
| **No regional restrictions** | Global Standard or Global Provisioned | Access to best available capacity |

**Capacity Planning Approach** (from [PTU onboarding guide](https://learn.microsoft.com/en-us/azure/ai-foundry/openai/how-to/provisioned-throughput-onboarding)):

To calculate and estimate your capacity requirements:

1. **Calculate your TPM requirements**: Determine required tokens per minute based on your expected workload
2. **Use the built-in capacity planner**: Available in Azure AI Foundry portal (Microsoft Foundry → Operate → Quota → Provisioned Throughput Unit tab)
3. **Input your metrics**: Enter input TPM and output TPM based on your workload characteristics
4. **Get PTU recommendation**: The calculator provides PTU allocation recommendation
5. **Compare costs**: Evaluate Standard (TPM) vs Provisioned (PTU) using the official pricing calculator

> **Note**: Microsoft does not publish specific "X requests/day = Y TPM" recommendations as capacity requirements vary significantly based on prompt size, response length, cache hit rates, and model choice. Use the built-in capacity planner with your actual workload characteristics.

# Error Resolution Workflows

**Table of Contents:** [Workflow 7: Quota Exhausted Recovery](#workflow-7-quota-exhausted-recovery) · [Workflow 8: Resolve 429 Rate Limit Errors](#workflow-8-resolve-429-rate-limit-errors) · [Workflow 9: Resolve DeploymentLimitReached](#workflow-9-resolve-deploymentlimitreached) · [Workflow 10: Resolve InsufficientQuota](#workflow-10-resolve-insufficientquota) · [Workflow 11: Resolve QuotaExceeded](#workflow-11-resolve-quotaexceeded)

## Workflow 7: Quota Exhausted Recovery

**A. Deploy to Different Region**
```bash
subId=$(az account show --query id -o tsv)
for region in eastus westus eastus2 westus2 swedencentral uksouth; do
  az rest --method get --url "https://management.azure.com/subscriptions/$subId/providers/Microsoft.CognitiveServices/locations/$region/usages?api-version=2023-05-01" \
    --query "value[?name.value=='OpenAI.Standard.gpt-4o'].{Model:name.value, Used:currentValue, Limit:limit, Available:(limit-currentValue)}" -o table &
done; wait
```

**B. Delete Unused Deployments**
```bash
az cognitiveservices account deployment delete --name <resource> --resource-group <rg> --deployment-name <deployment>
```

**C. Request Quota Increase (3-5 days)**

**D. Migrate to PTU** - See capacity-planning.md

---

## Workflow 8: Resolve 429 Rate Limit Errors

**Identify Deployment:**
```bash
az cognitiveservices account deployment list --name <resource> --resource-group <rg> \
  --query "[].{Name:name,Model:properties.model.name,TPM:sku.capacity*1000}" -o table
```

**Solutions:**

**A. Increase Capacity**
```bash
az cognitiveservices account deployment update --name <resource> --resource-group <rg> --deployment-name <deployment> --sku-capacity 100
```

**B. Add Retry Logic** - Exponential backoff in code

**C. Load Balance**
```bash
az cognitiveservices account deployment create --name <resource> --resource-group <rg> --deployment-name gpt-4o-2 \
  --model-name gpt-4o --model-version "2024-05-13" --model-format OpenAI --sku-name Standard --sku-capacity 100
```

**D. Migrate to PTU** - No rate limits

---

## Workflow 9: Resolve DeploymentLimitReached

**Root Cause:** 10-20 slots per resource.

**Check Count:**
```bash
deployment_count=$(az cognitiveservices account deployment list --name <resource> --resource-group <rg> --query "length(@)")
echo "Deployments: $deployment_count / ~20 slots"
```

**Find Test Deployments:**
```bash
az cognitiveservices account deployment list --name <resource> --resource-group <rg> \
  --query "[?contains(name,'test') || contains(name,'demo')].{Name:name}" -o table
```

**Delete:**
```bash
az cognitiveservices account deployment delete --name <resource> --resource-group <rg> --deployment-name <deployment>
```

**Or Create New Resource (fresh 10-20 slots):**
```bash
az cognitiveservices account create --name "my-foundry-2" --resource-group <rg> --location eastus --kind AIServices --sku S0 --yes
```

---

## Workflow 10: Resolve InsufficientQuota

**Root Cause:** Requested capacity exceeds available quota.

**Check Quota:**
```bash
subId=$(az account show --query id -o tsv)
az rest --method get --url "https://management.azure.com/subscriptions/$subId/providers/Microsoft.CognitiveServices/locations/eastus/usages?api-version=2023-05-01" \
  --query "value[?contains(name.value,'OpenAI')].{Model:name.value, Used:currentValue, Limit:limit, Available:(limit-currentValue)}" -o table
```

**Solutions:**

**A. Reduce Capacity**
```bash
az cognitiveservices account deployment create --name <resource> --resource-group <rg> --deployment-name gpt-4o \
  --model-name gpt-4o --model-version "2024-05-13" --model-format OpenAI --sku-name Standard --sku-capacity 20
```

**B. Delete Unused Deployments**
```bash
az cognitiveservices account deployment delete --name <resource> --resource-group <rg> --deployment-name <unused>
```

**C. Different Region** - Check quota with multi-region script (Workflow 7)

**D. Request Increase (3-5 days)**

---

## Workflow 11: Resolve QuotaExceeded

**Root Cause:** Deployment exceeds regional quota.

**Check Quota:**
```bash
subId=$(az account show --query id -o tsv)
az rest --method get --url "https://management.azure.com/subscriptions/$subId/providers/Microsoft.CognitiveServices/locations/eastus/usages?api-version=2023-05-01" \
  --query "value[?contains(name.value,'OpenAI')]" -o table
```

**Multi-Region Check:** (Use Workflow 7 script)

**Solutions:**

**A. Delete Unused Deployments**
```bash
az cognitiveservices account deployment delete --name <resource> --resource-group <rg> --deployment-name <unused>
```

**B. Different Region**
```bash
az cognitiveservices account deployment create --name <resource> --resource-group <rg> --deployment-name gpt-4o \
  --model-name gpt-4o --model-version "2024-05-13" --model-format OpenAI --sku-name Standard --sku-capacity 50
```

**C. Request Increase (3-5 days)**

**D. Reduce Capacity**

**Decision:** Available < 10% → Different region; 10-50% → Delete/reduce; > 50% → Delete one deployment

---

# Quota Optimization Strategies

Comprehensive strategies for optimizing Azure AI Foundry quota allocation and reducing costs.

**Table of Contents:** [1. Identify and Delete Unused Deployments](#1-identify-and-delete-unused-deployments) · [2. Right-Size Over-Provisioned Deployments](#2-right-size-over-provisioned-deployments) · [3. Consolidate Multiple Small Deployments](#3-consolidate-multiple-small-deployments) · [4. Cost Optimization Strategies](#4-cost-optimization-strategies) · [5. Regional Quota Rebalancing](#5-regional-quota-rebalancing)

## 1. Identify and Delete Unused Deployments

**Step 1: Discovery with Quota Context**

Get quota limits FIRST to understand how close you are to capacity:

```bash
# Check current quota usage vs limits (run this FIRST)
subId=$(az account show --query id -o tsv)
region="eastus"  # Change to your region
az rest --method get \
  --url "https://management.azure.com/subscriptions/$subId/providers/Microsoft.CognitiveServices/locations/$region/usages?api-version=2023-05-01" \
  --query "value[?contains(name.value,'OpenAI')].{Model:name.value, Used:currentValue, Limit:limit, Available:'(Limit - Used)'}" -o table
```

**Step 2: Parallel Deployment Enumeration**

List all deployments across resources efficiently:

```bash
# Get all Foundry resources
resources=$(az cognitiveservices account list --query "[?kind=='AIServices'].{name:name,rg:resourceGroup}" -o json)

# Parallel deployment enumeration (faster than sequential)
echo "$resources" | jq -r '.[] | "\(.name) \(.rg)"' | while read name rg; do
  echo "=== $name ($rg) ==="
  az cognitiveservices account deployment list --name "$name" --resource-group "$rg" \
    --query "[].{Deployment:name,Model:properties.model.name,Capacity:sku.capacity,Created:systemData.createdAt}" -o table &
done
wait  # Wait for all background jobs to complete
```

**Step 3: Identify Stale Deployments**

Criteria for deletion candidates:

- **Test/temporary naming**: Contains "test", "demo", "temp", "dev" in deployment name
- **Old timestamps**: Created >90 days ago with timestamp-based naming (e.g., "gpt4-20231015")
- **High capacity consumers**: Deployments with >100K TPM capacity that haven't been referenced in recent logs
- **Duplicate models**: Multiple deployments of same model/version in same region

**Example pattern matching for stale deployments:**
```bash
# Find deployments with test/temp naming
az cognitiveservices account deployment list --name <resource> --resource-group <rg> \
  --query "[?contains(name,'test') || contains(name,'demo') || contains(name,'temp')].{Name:name,Capacity:sku.capacity}" -o table
```

**Step 4: Delete and Verify Quota Recovery**

```bash
# Delete unused deployment (quota freed IMMEDIATELY)
az cognitiveservices account deployment delete --name <resource> --resource-group <rg> --deployment-name <deployment>

# Verify quota freed (re-run Step 1 quota check)
# You should see "Used" decrease by the deployment's capacity
```

**Cost Impact Analysis:**

| Deployment Type | Capacity (TPM) | Quota Freed | Cost Impact (TPM) | Cost Impact (PTU) |
|-----------------|----------------|-------------|-------------------|-------------------|
| Test deployment | 10K TPM | 10K TPM | $0 (pay-per-use) | N/A |
| Unused production | 100K TPM | 100K TPM | $0 (pay-per-use) | N/A |
| Abandoned PTU deployment | 100 PTU | ~40K TPM equivalent | $0 TPM | **$3,650/month saved** (100 PTU × 730h × $0.05/h) |
| High-capacity test | 450K TPM | 450K TPM | $0 (pay-per-use) | N/A |

**Key Insight:** For TPM (Standard) deployments, deletion frees quota but has no direct cost impact (you pay per token used). For PTU (Provisioned) deployments, deletion **immediately stops hourly charges** and can save thousands per month.

---

## 2. Right-Size Over-Provisioned Deployments

**Identify over-provisioned deployments:**
- Check Azure Monitor metrics for actual token usage
- Compare allocated TPM vs. peak usage
- Look for deployments with <50% utilization

**Right-sizing example:**
```bash
# Update deployment to lower capacity
az cognitiveservices account deployment update --name <resource> --resource-group <rg> \
  --deployment-name <deployment> --sku-capacity 30  # Reduce from 50K to 30K TPM
```

**Cost Optimization:**
- **TPM (Standard)**: Reduces regional quota consumption (no direct cost savings, pay-per-token)
- **PTU (Provisioned)**: Direct cost reduction (40% capacity reduction = 40% cost reduction)

---

## 3. Consolidate Multiple Small Deployments

**Pattern:** Multiple 10K TPM deployments → One 30-50K TPM deployment

**Benefits:**
- Fewer deployment slots consumed
- Simpler management
- Same total capacity, better utilization

**Example:**
- **Before**: 3 deployments @ 10K TPM each = 30K TPM total, 3 slots used
- **After**: 1 deployment @ 30K TPM = 30K TPM total, 1 slot used
- **Savings**: 2 deployment slots freed for other models

---

## 4. Cost Optimization Strategies

> **Official Documentation**: [Plan to manage costs for Azure OpenAI](https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/manage-costs) and [Fine-tuning cost management](https://learn.microsoft.com/en-us/azure/ai-foundry/openai/how-to/fine-tuning-cost-management)

**A. Use Fine-Tuned Smaller Models** (from [Microsoft Transparency Note](https://learn.microsoft.com/en-us/azure/ai-foundry/responsible-ai/openai/transparency-note)):

You can reduce costs or latency by swapping a fine-tuned version of a smaller/faster model (e.g., fine-tuned GPT-3.5-Turbo) for a more general-purpose model (e.g., GPT-4).

```bash
# Deploy fine-tuned GPT-3.5 Turbo as cost-effective alternative to GPT-4
az cognitiveservices account deployment create --name <resource> --resource-group <rg> \
  --deployment-name gpt-35-tuned --model-name <your-fine-tuned-model> \
  --model-format OpenAI --sku-name Standard --sku-capacity 10
```

**B. Remove Unused Fine-Tuned Deployments** (from [Fine-tuning cost management](https://learn.microsoft.com/en-us/azure/ai-foundry/openai/how-to/fine-tuning-cost-management)):

Fine-tuned model deployments incur **hourly hosting costs** even when not in use. Remove unused deployments promptly to control costs.

- Inactive deployments unused for **15 consecutive days** are automatically deleted
- Proactively delete unused fine-tuned deployments to avoid hourly charges

```bash
# Delete unused fine-tuned deployment
az cognitiveservices account deployment delete --name <resource> --resource-group <rg> \
  --deployment-name <unused-fine-tuned-deployment>
```

**C. Batch Multiple Requests** (from [Cost optimization Q&A](https://learn.microsoft.com/en-us/answers/questions/1689253/how-to-optimize-costs-per-request-azure-openai-gpt)):

Batch multiple requests together to reduce the total number of API calls and lower overall costs.

**D. Use Commitment Tiers for Predictable Costs** (from [Managing costs guide](https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/manage-costs)):

- **Pay-as-you-go**: Bills according to usage (variable costs)
- **Commitment tiers**: Commit to using service features for a fixed fee (predictable costs, potential savings for consistent usage)

---

## 5. Regional Quota Rebalancing

If you have quota spread across multiple regions but only use some:

```bash
# Check quota across regions
for region in eastus westus uksouth; do
  echo "=== $region ==="
  subId=$(az account show --query id -o tsv)
  az rest --method get \
    --url "https://management.azure.com/subscriptions/$subId/providers/Microsoft.CognitiveServices/locations/$region/usages?api-version=2023-05-01" \
    --query "value[?contains(name.value,'OpenAI')].{Model:name.value, Used:currentValue, Limit:limit}" -o table
done
```

**Optimization:** Concentrate deployments in fewer regions to maximize quota utilization per region.

# Provisioned Throughput Units (PTU) Guide

**Table of Contents:** [Understanding PTU vs Standard TPM](#understanding-ptu-vs-standard-tpm) · [When to Use PTU](#when-to-use-ptu) · [PTU Capacity Planning](#ptu-capacity-planning) · [Deploy Model with PTU](#deploy-model-with-ptu) · [Request PTU Quota Increase](#request-ptu-quota-increase) · [Understanding Region and Deployment Quotas](#understanding-region-and-deployment-quotas) · [External Resources](#external-resources)

## Understanding PTU vs Standard TPM

Microsoft Foundry offers two quota types:

### Standard TPM (Tokens Per Minute)
- Pay-as-you-go model, charged per token
- Each deployment consumes capacity units (e.g., 10K TPM, 50K TPM)
- Total regional quota shared across all deployments
- Subject to rate limiting during high demand (429 errors possible)
- Best for: Variable workloads, development, testing, bursty traffic

### Provisioned Throughput Units (PTU)
- Monthly commitment for guaranteed throughput
- No rate limiting, consistent latency
- Measured in PTU units (not TPM)
- Best for: Predictable, high-volume production workloads
- More cost-effective when consistent token usage justifies monthly commitment

## When to Use PTU

| Factor | Standard (TPM) | Provisioned (PTU) |
|--------|----------------|-------------------|
| **Best For** | Variable workloads, development, testing | Predictable production workloads |
| **Pricing** | Pay-per-token | Monthly commitment (hourly rate per PTU) |
| **Rate Limits** | Yes (429 errors possible) | No (guaranteed throughput) |
| **Latency** | Variable | Consistent |
| **Cost Decision** | Lower upfront commitment | More economical for consistent, high-volume usage |
| **Flexibility** | Scale up/down instantly | Requires planning and commitment |
| **Use Case** | Prototyping, bursty traffic | Production apps, high-volume APIs |

**Use PTU when:**
- Consistent, predictable token usage where monthly commitment is cost-effective
- Need guaranteed throughput (no 429 rate limit errors)
- Require consistent latency with performance SLA
- High-volume production workloads with stable traffic patterns

**Decision Guidance:**
Compare your current pay-as-you-go costs with PTU pricing. PTU may be more economical when consistent usage justifies the monthly commitment.

## PTU Capacity Planning

### Official Calculation Methods

> **Agent Instruction:** Only present official Azure capacity calculator methods below. Do NOT generate or suggest estimated PTU formulas, TPM-per-PTU conversion tables, or reference deprecated calculators (oai.azure.com/portal/calculator).

Calculate PTU requirements using these official methods:

**Method 1: Microsoft Foundry Portal**
1. Navigate to Microsoft Foundry portal
2. Go to **Operate** → **Quota**
3. Select **Provisioned throughput unit** tab
4. Click **Capacity calculator** button
5. Enter workload parameters (model, tokens/call, RPM, latency target)
6. Calculator returns exact PTU count needed

**Method 2: Using Azure REST API**
```bash
# Calculate required PTU capacity
curl -X POST "https://management.azure.com/subscriptions/<subscription-id>/providers/Microsoft.CognitiveServices/calculateModelCapacity?api-version=2024-10-01" \
  -H "Authorization: Bearer <access-token>" \
  -H "Content-Type: application/json" \
  -d '{
    "model": {
      "format": "OpenAI",
      "name": "gpt-4o",
      "version": "2024-05-13"
    },
    "workload": {
      "requestPerMin": 100,
      "tokensPerMin": 50000,
      "peakRequestsPerMin": 150
    }
  }'
```

## Deploy Model with PTU

### Step 1: Calculate PTU Requirements

Use the official capacity calculator methods above to determine required PTU capacity.

### Step 2: Deploy with PTU

```bash
# Deploy model with calculated PTU capacity
az cognitiveservices account deployment create \
  --name <resource-name> \
  --resource-group <rg> \
  --deployment-name gpt-4o-ptu-deployment \
  --model-name gpt-4o \
  --model-version "2024-05-13" \
  --model-format OpenAI \
  --sku-name ProvisionedManaged \
  --sku-capacity 100

# Check PTU deployment status
az cognitiveservices account deployment show \
  --name <resource-name> \
  --resource-group <rg> \
  --deployment-name gpt-4o-ptu-deployment
```

**Key Differences from Standard TPM:**
- SKU name: `ProvisionedManaged` (not `Standard`)
- Capacity: Measured in PTU units (not K TPM)
- Billing: Monthly commitment regardless of usage
- No rate limiting (guaranteed throughput)

## Request PTU Quota Increase

PTU quota is separate from TPM quota and requires specific justification:

1. Navigate to Azure Portal → Foundry resource → **Quotas**
2. Select **Provisioned throughput unit** tab
3. Identify model needing PTU increase (e.g., "GPT-4o PTU")
4. Click **Request quota increase**
5. Fill form:
   - Model name
   - Requested PTU quota
   - Include capacity calculator results in business justification
   - Explain workload characteristics (volume, latency requirements)
6. Submit and monitor status

**Processing Time:** Typically 3-5 business days (longer than standard quota requests)
**Note:** PTU quota requests typically require stronger business justification due to commitment nature

**Alternative:** Deploy to different region with available PTU quota

## Understanding Region and Deployment Quotas

### Region Quota
- Maximum PTU capacity available in an Azure region
- Varies by model type (GPT-4, GPT-4o, etc.)
- Shared across subscription resources in same region
- Separate from TPM quota (you have both TPM and PTU quotas)

### Deployment Slots
- Number of concurrent model deployments allowed
- Typically 10-20 slots per resource
- Each PTU deployment uses one slot (same as TPM deployments)
- Deployment count limit is independent of capacity

## External Resources

- [Understanding PTU Costs](https://learn.microsoft.com/azure/ai-foundry/openai/how-to/provisioned-throughput-onboarding)
- [What Is Provisioned Throughput](https://learn.microsoft.com/azure/ai-foundry/openai/concepts/provisioned-throughput)
- [Calculate Model Capacity API](https://learn.microsoft.com/rest/api/aiservices/accountmanagement/calculate-model-capacity/calculate-model-capacity?view=rest-aiservices-accountmanagement-2024-10-01&tabs=HTTP)
- [PTU Overview](https://learn.microsoft.com/azure/ai-services/openai/concepts/provisioned-throughput)

# Troubleshooting Quota Errors

**Table of Contents:** [Common Quota Errors](#common-quota-errors) · [Detailed Error Resolution](#detailed-error-resolution) · [Request Quota Increase Process](#request-quota-increase-process) · [Diagnostic Commands](#diagnostic-commands) · [External Resources](#external-resources)

## Common Quota Errors

| Error | Cause | Quick Fix |
|-------|-------|-----------|
| `QuotaExceeded` | Regional quota consumed (TPM or PTU) | Delete unused deployments or request increase |
| `InsufficientQuota` | Not enough available for requested capacity | Reduce deployment capacity or free quota |
| `DeploymentLimitReached` | Too many deployment slots used | Delete unused deployments to free slots |
| `429 Rate Limit` | TPM capacity too low for traffic (Standard only) | Increase TPM capacity or migrate to PTU |
| `PTU capacity unavailable` | No PTU quota in region | Request PTU quota or try different region |
| `SKU not supported` | PTU not available for model/region | Check model availability or use Standard TPM |

## Detailed Error Resolution

### QuotaExceeded Error

All available TPM or PTU quota consumed in the region.

**Resolution:**

1. **Check current quota usage:**
   ```bash
   subId=$(az account show --query id -o tsv)
   region="eastus"
   az rest --method get \
     --url "https://management.azure.com/subscriptions/$subId/providers/Microsoft.CognitiveServices/locations/$region/usages?api-version=2023-05-01" \
     --query "value[?contains(name.value,'OpenAI')].{Model:name.value, Used:currentValue, Limit:limit}" -o table
   ```

2. **Choose resolution:**
   - **Option A**: Delete unused deployments to free quota
   - **Option B**: Reduce requested deployment capacity
   - **Option C**: Deploy to different region with available quota
   - **Option D**: Request quota increase through Azure Portal

### InsufficientQuota Error

Available quota less than requested capacity.

**Resolution:**

1. **Check available quota:**
   ```bash
   # Calculate available: limit - currentValue
   subId=$(az account show --query id -o tsv)
   region="eastus"
   az rest --method get \
     --url "https://management.azure.com/subscriptions/$subId/providers/Microsoft.CognitiveServices/locations/$region/usages?api-version=2023-05-01" \
     --query "value[?name.value=='OpenAI.Standard.gpt-4o'].{Model:name.value, Used:currentValue, Limit:limit, Available:(limit-currentValue)}" -o table
   ```

2. **Options:**
   - Reduce deployment capacity to fit available quota
   - Delete existing deployments to free capacity
   - Try different region with more available quota
   - Request quota increase

### DeploymentLimitReached Error

Resource reached maximum deployment slot limit (10-20 slots).

**Resolution:**

1. **List existing deployments:**
   ```bash
   az cognitiveservices account deployment list \
     --name <resource-name> \
     --resource-group <rg> \
     --query '[].{Name:name, Model:properties.model.name, Capacity:sku.capacity}' \
     --output table
   ```

2. **Delete unused deployments:**
   ```bash
   az cognitiveservices account deployment delete \
     --name <resource-name> \
     --resource-group <rg> \
     --deployment-name <unused-deployment-name>
   ```

3. **Verify slot freed:**
   ```bash
   az cognitiveservices account deployment list \
     --name <resource-name> \
     --resource-group <rg> \
     --query 'length([])'
   ```

### 429 Rate Limit Errors

TPM capacity insufficient for traffic volume (Standard TPM only).

**Resolution:**

1. **Check deployment capacity:**
   ```bash
   az cognitiveservices account deployment show \
     --name <resource-name> \
     --resource-group <rg> \
     --deployment-name <deployment-name> \
     --query '{Name:name, Model:properties.model.name, Capacity:sku.capacity, SKU:sku.name}'
   ```

2. **Options:**
   - **Option A**: Increase TPM capacity on existing deployment
     ```bash
     az cognitiveservices account deployment update \
       --name <resource-name> \
       --resource-group <rg> \
       --deployment-name <deployment-name> \
       --sku-capacity <higher-capacity>
     ```
   - **Option B**: Migrate to PTU for guaranteed throughput (no rate limits)
   - **Option C**: Implement retry logic with exponential backoff in application

### PTU Capacity Unavailable Error

No PTU quota allocated in region, or PTU not available for model/region.

**Resolution:**

1. **Check PTU quota:**
   ```bash
   subId=$(az account show --query id -o tsv)
   region="eastus"
   az rest --method get \
     --url "https://management.azure.com/subscriptions/$subId/providers/Microsoft.CognitiveServices/locations/$region/usages?api-version=2023-05-01" \
     --query "value[?contains(name.value,'ProvisionedManaged')].{Model:name.value, Used:currentValue, Limit:limit}" -o table
   ```

2. **Options:**
   - Request PTU quota increase through Azure Portal (include capacity calculator results)
   - Try different region where PTU is available
   - Use Standard TPM instead

### SKU Not Supported Error

PTU not available for specific model or region combination.

**Resolution:**

1. **Check model availability:**
   - Review [PTU model availability by region](https://learn.microsoft.com/azure/ai-services/openai/concepts/models#provisioned-deployment-model-availability)

2. **Options:**
   - Deploy with Standard TPM SKU instead
   - Choose different region where PTU is supported
   - Use alternative model that supports PTU in your region

## Request Quota Increase Process

### For Standard TPM Quota

1. Navigate to Azure Portal → Your Foundry resource → **Quotas**
2. Identify model needing increase (e.g., "GPT-4o Standard")
3. Click **Request quota increase**
4. Fill form:
   - Model name
   - Requested quota (in TPM)
   - Business justification (required)
5. Submit and monitor status

**Processing Time:** Typically 1-2 business days

### For PTU Quota

1. Navigate to Azure Portal → Your Foundry resource → **Quotas**
2. Select **Provisioned throughput unit** tab
3. Identify model needing PTU increase
4. Click **Request quota increase**
5. Fill form:
   - Model name
   - Requested PTU quota
   - Include capacity calculator results
   - Detailed business justification (workload characteristics)
6. Submit and monitor status

**Processing Time:** Typically 3-5 business days (requires stronger justification)

## Diagnostic Commands

```bash
# Check deployment status
az cognitiveservices account deployment show \
  --name <resource-name> \
  --resource-group <rg> \
  --deployment-name <deployment-name>

# Verify available quota
subId=$(az account show --query id -o tsv)
az rest --method get \
  --url "https://management.azure.com/subscriptions/$subId/providers/Microsoft.CognitiveServices/locations/eastus/usages?api-version=2023-05-01" \
  --query "value[?contains(name.value,'OpenAI')].{Model:name.value, Used:currentValue, Limit:limit, Available:(limit-currentValue)}" \
  --output table

# List all deployments
az cognitiveservices account deployment list \
  --name <resource-name> \
  --resource-group <rg> \
  --query '[].{Name:name, Model:properties.model.name, Capacity:sku.capacity, SKU:sku.name}' \
  --output table
```

## External Resources

- [Quota Management Documentation](https://learn.microsoft.com/azure/ai-services/openai/how-to/quota)
- [Rate Limits Documentation](https://learn.microsoft.com/azure/ai-services/openai/quotas-limits)
- [Troubleshooting Guide](https://learn.microsoft.com/azure/ai-services/openai/troubleshooting)

# Detailed Workflows: Quota Management

**Table of Contents:** [Workflow 1: View Current Quota Usage](#workflow-1-view-current-quota-usage---detailed-steps) · [Workflow 2: Find Best Region for Model Deployment](#workflow-2-find-best-region-for-model-deployment---detailed-steps) · [Workflow 3: Check Quota Before Deployment](#workflow-3-check-quota-before-deployment---detailed-steps) · [Workflow 4: Monitor Quota Across Deployments](#workflow-4-monitor-quota-across-deployments---detailed-steps) · [Quick Command Reference](#quick-command-reference) · [MCP Tools Reference](#mcp-tools-reference-optional-wrappers)

## Workflow 1: View Current Quota Usage - Detailed Steps

### Step 1: Show Regional Quota Summary (REQUIRED APPROACH)

> **CRITICAL AGENT INSTRUCTION:**
> - When showing quota: Query REGIONAL quota summary, NOT individual resources
> - DO NOT run `az cognitiveservices account list` for quota queries
> - DO NOT filter resources by username or name patterns
> - ONLY check specific resource deployments if user provides resource name
> - Quotas are managed at SUBSCRIPTION + REGION level, NOT per-resource

**Show Regional Quota Summary:**

```bash
# Get subscription ID
subId=$(az account show --query id -o tsv)

# Check quota for key regions
regions=("eastus" "eastus2" "westus" "westus2")
for region in "${regions[@]}"; do
  echo "=== Region: $region ==="
  az rest --method get \
    --url "https://management.azure.com/subscriptions/$subId/providers/Microsoft.CognitiveServices/locations/$region/usages?api-version=2023-05-01" \
    --query "value[?contains(name.value,'OpenAI.Standard')].{Model:name.value, Used:currentValue, Limit:limit, Available:(limit-currentValue)}" \
    --output table
  echo ""
done
```

### Step 2: If User Asks for Specific Resource (ONLY IF EXPLICITLY REQUESTED)

```bash
# User must provide resource name
az cognitiveservices account deployment list \
  --name <user-provided-resource-name> \
  --resource-group <user-provided-rg> \
  --query '[].{Name:name, Model:properties.model.name, Capacity:sku.capacity, SKU:sku.name}' \
  --output table
```

**Alternative - Use MCP Tools (Optional Wrappers):**
```
foundry_models_deployments_list(
  resource-group="<rg>",
  azure-ai-services="<resource-name>"
)
```
*Note: MCP tools are convenience wrappers around the same control plane APIs shown above.*

**Interpreting Results:**
- `Used` (currentValue): Currently allocated quota
- `Limit`: Maximum quota available in region
- `Available`: Calculated as `limit - currentValue`

## Workflow 2: Find Best Region for Model Deployment - Detailed Steps

### Step 1: Check Single Region

```bash
# Get subscription ID
subId=$(az account show --query id -o tsv)

# Check quota for GPT-4o Standard in a specific region
region="eastus"  # Change to your target region
az rest --method get \
  --url "https://management.azure.com/subscriptions/$subId/providers/Microsoft.CognitiveServices/locations/$region/usages?api-version=2023-05-01" \
  --query "value[?name.value=='OpenAI.Standard.gpt-4o'].{Model:name.value, Used:currentValue, Limit:limit, Available:(limit-currentValue)}" \
  -o table
```

### Step 2: Check Multiple Regions (Common Regions)

Check these regions in sequence by changing the `region` variable:
- `eastus`, `eastus2` - US East Coast
- `westus`, `westus2`, `westus3` - US West Coast
- `swedencentral` - Europe (Sweden)
- `canadacentral` - Canada
- `uksouth` - UK
- `japaneast` - Asia Pacific

**Alternative - Use MCP Tool:**
```
model_quota_list(region="eastus")
```
Repeat for each target region.

**Key Points:**
- Query returns `currentValue` (used), `limit` (max), and calculated `Available`
- Standard SKU format: `OpenAI.Standard.<model-name>`
- For PTU: `OpenAI.ProvisionedManaged.<model-name>`
- Focus on 2-3 regions relevant to your location rather than checking all regions

## Workflow 3: Check Quota Before Deployment - Detailed Steps

**Steps:**
1. Check current usage (workflow #1)
2. Calculate available: `limit - currentValue`
3. Compare: `available >= required_capacity`
4. If insufficient: Use workflow #2 to find region with capacity, or request increase

## Workflow 4: Monitor Quota Across Deployments - Detailed Steps

**Recommended Approach - Regional Quota Overview:**

Show quota by region (better than listing all resources):

```bash
subId=$(az account show --query id -o tsv)
regions=("eastus" "eastus2" "westus" "westus2" "swedencentral")

for region in "${regions[@]}"; do
  echo "=== Region: $region ==="
  az rest --method get \
    --url "https://management.azure.com/subscriptions/$subId/providers/Microsoft.CognitiveServices/locations/$region/usages?api-version=2023-05-01" \
    --query "value[?contains(name.value,'OpenAI')].{Model:name.value, Used:currentValue, Limit:limit, Available:(limit-currentValue)}" \
    --output table
  echo ""
done
```

**Alternative - Check Specific Resource:**

If user wants to monitor a specific resource, ask for resource name first:

```bash
# List deployments for specific resource
az cognitiveservices account deployment list \
  --name <resource-name> \
  --resource-group <rg> \
  --query '[].{Name:name, Model:properties.model.name, Capacity:sku.capacity}' \
  --output table
```

> **Note:** Don't automatically iterate through all resources in the subscription. Show regional quota summary or ask for specific resource name.

## Quick Command Reference

```bash
# View quota for specific model using REST API
subId=$(az account show --query id -o tsv)
region="eastus"  # Change to your region
az rest --method get \
  --url "https://management.azure.com/subscriptions/$subId/providers/Microsoft.CognitiveServices/locations/$region/usages?api-version=2023-05-01" \
  --query "value[?contains(name.value,'gpt-4')].{Name:name.value, Used:currentValue, Limit:limit, Available:(limit-currentValue)}" \
  --output table

# List all deployments with capacity
az cognitiveservices account deployment list \
  --name <resource-name> \
  --resource-group <rg> \
  --query '[].{Name:name, Model:properties.model.name, Capacity:sku.capacity}' \
  --output table

# Delete deployment to free quota
az cognitiveservices account deployment delete \
  --name <resource-name> \
  --resource-group <rg> \
  --deployment-name <deployment-name>
```

## MCP Tools Reference (Optional Wrappers)

**Note:** All quota operations are control plane (management) operations. MCP tools are optional convenience wrappers around Azure CLI commands.

| Tool | Purpose | Equivalent Azure CLI |
|------|---------|---------------------|
| `foundry_models_deployments_list` | List all deployments with capacity | `az cognitiveservices account deployment list` |
| `model_quota_list` | List quota and usage across regions | `az rest` (Management API) |
| `model_catalog_list` | List available models from catalog | `az rest` (Management API) |
| `foundry_resource_get` | Get resource details and endpoint | `az cognitiveservices account show` |

**Recommended:** Use Azure CLI commands directly for control plane operations.

# Microsoft Foundry RBAC Management

Reference for managing RBAC for Microsoft Foundry resources: user permissions, managed identity configuration, and service principal setup for CI/CD.

## Quick Reference

| Property | Value |
|----------|-------|
| **CLI Extension** | `az role assignment`, `az ad sp` |
| **Resource Type** | `Microsoft.CognitiveServices/accounts` |
| **Best For** | Permission management, access auditing, CI/CD setup |

## When to Use

- Grant user access to Foundry resources or projects
- Set up developer permissions (Project Manager, Owner roles)
- Audit role assignments or validate permissions
- Configure managed identity roles for connected resources
- Create service principals for CI/CD pipeline automation
- Troubleshoot permission errors

## Foundry Built-in Roles

| Role | Create Projects | Data Actions | Role Assignments |
|------|-----------------|--------------|------------------|
| Foundry User | No | Yes | No |
| Foundry Project Manager | Yes | Yes | Yes (Foundry User only) |
| Foundry Account Owner | Yes | No | Yes (Foundry User only) |
| Foundry Owner | Yes | Yes | Yes |

> ⚠️ **Warning:** Foundry User is auto-assigned via Portal but NOT via SDK/CLI. Automation must explicitly assign roles.

## Workflows

All scopes follow the pattern: `/subscriptions/<subscription-id>/resourceGroups/<resource-group>/providers/Microsoft.CognitiveServices/accounts/<foundry-resource-name>`

For project-level scoping, append `/projects/<project-name>`.

### 1. Assign User Permissions

```bash
az role assignment create --role "53ca6127-db72-4b80-b1b0-d745d6d5456d" --assignee "<user-email-or-object-id>" --scope "<foundry-scope>" # Foundry User
```

### 2. Assign Developer Permissions

```bash
# Project Manager (create projects, assign Foundry User roles)
az role assignment create --role "eadc314b-1a2d-4efa-be10-5d325db5065e" --assignee "<user-email-or-object-id>" --scope "<foundry-scope>" # Foundry Project Manager

# Full ownership including data actions
az role assignment create --role "c883944f-8b7b-4483-af10-35834be79c4a" --assignee "<user-email-or-object-id>" --scope "<foundry-scope>" # Foundry Owner
```

### 3. Audit Role Assignments

```bash
# List all assignments
az role assignment list --scope "<foundry-scope>" --output table

# Detailed with principal names
az role assignment list --scope "<foundry-scope>" --query "[].{Principal:principalName, PrincipalType:principalType, Role:roleDefinitionName}" --output table

# Foundry roles only
az role assignment list --scope "<foundry-scope>" --query "[?contains(roleDefinitionName, 'Foundry')].{Principal:principalName, Role:roleDefinitionName}" --output table
```

### 4. Validate Permissions

```bash
# Current user's roles on resource
az role assignment list --assignee "$(az ad signed-in-user show --query id -o tsv)" --scope "<foundry-scope>" --query "[].roleDefinitionName" --output tsv

# Check actions available to a role
az role definition list --name "Foundry User" --query "[].permissions[].actions" --output json
```

**Permission Requirements by Action:**

| Action | Required Role(s) |
|--------|------------------|
| Deploy models | Foundry User, Foundry Project Manager, Foundry Owner |
| Create projects | Foundry Project Manager, Foundry Account Owner, Foundry Owner |
| Assign Foundry User role | Foundry Project Manager, Foundry Account Owner, Foundry Owner |
| Full data access | Foundry User, Foundry Project Manager, Foundry Owner |

### 5. Configure Managed Identity Roles

```bash
# Get managed identity principal ID
PRINCIPAL_ID=$(az cognitiveservices account show --name <foundry-resource-name> --resource-group <resource-group> --query identity.principalId --output tsv)

# Assign roles to connected resources (repeat pattern for each)
az role assignment create --role "<role-name>" --assignee "$PRINCIPAL_ID" --scope "<resource-scope>"
```

**Common Managed Identity Role Assignments:**

| Connected Resource | Role | Purpose |
|--------------------|------|---------|
| Azure Storage | Storage Blob Data Reader | Read files/documents |
| Azure Storage | Storage Blob Data Contributor | Read/write files |
| Azure Key Vault | Key Vault Secrets User | Read secrets |
| Azure AI Search | Search Index Data Reader | Query indexes |
| Azure AI Search | Search Index Data Contributor | Query and modify indexes |
| Azure Cosmos DB | Cosmos DB Account Reader | Read data |

### 6. Create Service Principal for CI/CD

```bash
# Create SP with minimal role
az ad sp create-for-rbac --name "foundry-cicd-sp" --role "53ca6127-db72-4b80-b1b0-d745d6d5456d" --scopes "<foundry-scope>" --output json # Foundry User
# Output contains: appId, password, tenant — store securely

# For project management permissions
az ad sp create-for-rbac --name "foundry-cicd-admin-sp" --role "eadc314b-1a2d-4efa-be10-5d325db5065e" --scopes "<foundry-scope>" --output json # Foundry Project Manager

# Add Contributor for resource provisioning
SP_APP_ID=$(az ad sp list --display-name "foundry-cicd-sp" --query "[0].appId" -o tsv)
az role assignment create --role "Contributor" --assignee "$SP_APP_ID" --scope "/subscriptions/<subscription-id>/resourceGroups/<resource-group>"
```

> 💡 **Tip:** Use least privilege — start with `Foundry User` and add roles as needed.

| CI/CD Scenario | Recommended Role | Additional Roles |
|----------------|------------------|------------------|
| Deploy models only | Foundry User | None |
| Manage projects | Foundry Project Manager | None |
| Full provisioning | Foundry Owner | Contributor (on RG) |
| Read-only monitoring | Reader | Foundry User (for data) |

**CI/CD Pipeline Login:**

```bash
az login --service-principal --username "<app-id>" --password "<client-secret>" --tenant "<tenant-id>"
az account set --subscription "<subscription-id>"
```

## Error Handling

| Issue | Cause | Resolution |
|-------|-------|------------|
| "Authorization failed" when deploying | Missing Foundry User role | Assign Foundry User role at resource scope |
| Cannot create projects | Missing Project Manager or Owner role | Assign Foundry Project Manager role |
| "Access denied" on connected resources | Managed identity missing roles | Assign appropriate roles to MI on each resource |
| Portal works but CLI fails | Portal auto-assigns roles, CLI doesn't | Explicitly assign Foundry User via CLI |
| Service principal cannot access data | Wrong role or scope | Verify Foundry User is assigned at correct scope |
| "Principal does not exist" | User/SP not found in directory | Verify the assignee email or object ID is correct |
| Role assignment already exists | Duplicate assignment attempt | Use `az role assignment list` to verify existing assignments |

## Additional Resources

- [Azure AI Foundry RBAC Documentation](https://learn.microsoft.com/azure/ai-foundry/concepts/rbac-ai-foundry)
- [Azure Built-in Roles](https://learn.microsoft.com/azure/role-based-access-control/built-in-roles)
- [Managed Identities Overview](https://learn.microsoft.com/azure/active-directory/managed-identities-azure-resources/overview)
- [Service Principal Authentication](https://learn.microsoft.com/azure/developer/github/connect-from-azure)

# Agent Metadata Contract

Use this contract for Microsoft Foundry agent folders. In azd projects, `.foundry/agent-metadata*.yaml` is an overlay/cache, not the source of truth for azd-owned deployment context.

## Local Layout

```text
<agent-root>/
  .foundry/
    agent-metadata.yaml
    agent-metadata.<env>.yaml
    suites/
    datasets/
    evaluators/
    results/
```

- `agent-metadata.yaml` is the preferred local/dev overlay.
- Optional `agent-metadata.<env>.yaml` files can hold a single prod or CI-targeted overlay.
- `suites/`, `datasets/`, `evaluators/`, and `results/` are local cache/result folders. Ask before overwriting user-edited files.

## Effective Context Model

Resolve deployment and evaluation context by layering sources in this order:

| Value | Preferred source | Fallbacks | Metadata write behavior |
|-------|------------------|-----------|-------------------------|
| Agent root | `azure.yaml` service `project` for `host: azure.ai.agent` | `.foundry` discovery, user path | Do not write except to initialize cache |
| Environment | user/session, then azd env/default | metadata `defaultEnvironment` | Store azd binding only when useful |
| Project endpoint | `azd env get-values` | metadata, user input | Do not duplicate azd values |
| Agent name/version | azd `AGENT_<SERVICE>_*` vars | `azure.yaml`, metadata, user input | Do not duplicate azd values |
| ACR | azd registry vars | metadata, user input | Do not duplicate azd values |
| Observability | azd App Insights vars | metadata, user input | Do not copy secrets if azd has them |
| Local eval draft | `eval.yaml` | metadata, user input | Sync to `.foundry` only after remote lookup/registration |
| Remote suite/cache refs | metadata | Foundry lookups | Persist in `.foundry` |

If azd and metadata both provide the same value and differ, stop and ask which source is authoritative. If they match, use the azd value and omit the duplicate on future metadata rewrites.

## Environment Overlay Model

| Field | Required when | Purpose |
|-------|---------------|---------|
| `defaultEnvironment` | Any metadata file exists | Default key inside this overlay file |
| `environments.<env>.azd.environmentName` | Optional | Binds overlay to an azd environment |
| `environments.<env>.azd.service` | Optional | Binds overlay to an `azure.yaml` service |
| `environments.<env>.projectEndpoint` | Required for non-azd/manual workflows | Explicit override when azd cannot resolve it |
| `environments.<env>.agentName` / `agentVersion` | `agentName` required for non-azd/manual workflows; `agentVersion` optional | Explicit override when azd cannot resolve it |
| `environments.<env>.azureContainerRegistry` | Required for non-azd/manual hosted-agent Docker/ACR deploy flow | Explicit override when azd cannot resolve it |
| `environments.<env>.observability.*` | Required only for trace workflows when azd cannot resolve observability | Trace lookup config when azd cannot resolve it |
| `environments.<env>.evaluationSuites[]` | Required after evaluation setup/sync | Remote suite/dataset/evaluator refs plus local cache paths |
| `environments.<env>.lastEval` | Optional | Last local result summary and result file path |

## Example azd Overlay

```yaml
defaultEnvironment: dev
environments:
  dev:
    azd:
      environmentName: <azd-env-name>
      service: <azure-yaml-service-name>
    evaluationSuites:
      - id: smoke-core
        suiteName: <foundry-suite-name>
        suiteVersion: "1"
        generationSource: eval-yaml
        tags:
          tier: smoke
          purpose: baseline
        suiteFile: .foundry/suites/<suite>-v1.json
        dataset: <dataset-name>
        datasetVersion: "1"
        datasetFile: .foundry/datasets/<agent>-<dataset>-v1.ref.json
        datasetUri: <foundry-dataset-uri>
        evaluators:
          - name: <evaluator-name>
            version: "1"
            threshold: 4
            definitionFile: .foundry/evaluators/<evaluator>-v1.json
```

## Example Manual Overlay

```yaml
defaultEnvironment: dev
environments:
  dev:
    projectEndpoint: https://<account>.services.ai.azure.com/api/projects/<project>
    agentName: <agent-name>
    azureContainerRegistry: <registry>.azurecr.io
    evaluationSuites:
      - id: smoke-core
        datasetFile: .foundry/datasets/<agent>-smoke-v1.ref.json
        evaluators:
          - name: relevance
            threshold: 4
```

## eval.yaml Mapping

When `eval.yaml` exists in the selected agent root, treat it as local evaluation intent, not proof of a Foundry suite.

| eval.yaml field | Use |
|-----------------|-----|
| `agent.name` | Candidate target agent; verify it matches selected context |
| `dataset.local_uri` | Local seed dataset candidate |
| `dataset.name`, `dataset.version` | Registered dataset candidate |
| `validation_dataset` | Optional validation dataset candidate |
| `evaluators[]` | Candidate evaluator names; verify with `evaluator_catalog_get` |
| `name` | Candidate eval/suite name; verify remotely before storing as `suiteName` |
| `options.eval_model` | Candidate judge/generation deployment |
| `options.optimization_model` | Candidate optimizer reasoning deployment |
| `options.max_candidates` | Candidate optimization iteration limit |
| `options.optimization_config.model_search_space` | Candidate target model search space |
| `options.pass_threshold` | Candidate evaluator threshold/default pass gate |
| `max_samples`, `trace_days`, `generation_instruction` | Suite setup defaults |

Legacy `dataset_file`, `dataset_reference`, and `validation_reference` keys may be normalized in memory when reading older files, but new files should use `dataset` and `validation_dataset`.

Persist eval.yaml-derived suite metadata only after the relevant dataset/evaluator/suite has been registered or found in Foundry. Use `generationSource: eval-yaml` for synced suite entries created from local eval config.

## Workflow Rules

1. Prefer azd service discovery before `.foundry` discovery when `azure.yaml` has `host: azure.ai.agent`.
2. Once an agent root is selected, use only that root's `.foundry`, source tree, `azure.yaml`, and `eval.yaml` unless the user switches roots.
3. Select metadata files in this order: explicit file/path, environment sidecar, `.foundry/agent-metadata.yaml`, then prompt if ambiguous.
4. Resolve environment from user/session, azd env/default, single-environment metadata, then `defaultEnvironment`.
5. Keep the selected root, environment, metadata overlay file, and primary context source visible in deploy/eval/trace summaries.
6. Treat metadata deployment fields as overrides when azd cannot resolve the value.
7. Treat `evaluationSuites[]` as the canonical synced suite model; normalize legacy fields in memory before use.
8. Writes target only the selected metadata file and selected environment. Never merge sibling metadata files automatically.
9. On metadata rewrites for azd projects, persist non-derivable overlay/cache state and omit azd-owned deployment duplicates.
10. Never silently overwrite cache files or metadata. Show a summary before refreshing, pruning duplicate fields, or replacing suite refs.

## Legacy Compatibility

If the selected environment has `testSuites[]` but no `evaluationSuites[]`, treat `testSuites[]` as the current suite source and migrate it on the next metadata write. If it has only legacy `testCases[]`, normalize that list the same way.

Preserve `id`, `suiteName`, `suiteVersion`, `generationJobId`, `generationSource`, `dataset`, `datasetVersion`, `datasetFile`, `datasetUri`, `evaluators`, and existing `tags`. Map legacy `priority` to `tags.tier` only when `tags.tier` is missing: `P0` -> `smoke`, `P1` -> `regression`, `P2` -> `coverage`.

## Evaluation Suite Guidance

Use `tags` as freeform key/value metadata. Suggested keys: `tier` (`smoke`, `regression`, `coverage`), `purpose` (`baseline`, `safety`, `tools`, `quality`), and `stage` (`local`, `generated`, `traces`, `curated`, `prod`).

Each synced suite should point to one dataset and one or more evaluators with thresholds. Store stable remote names separately from versions, keep local cache filenames versioned, and persist `suiteFile`, `datasetFile`, `datasetContentPath`, `datasetUri`, and evaluator `definitionFile` when available. Local dataset filenames should start with the effective Foundry agent name. Use evaluation-suite IDs in evaluation names, result folders, and regression summaries.

For generated Foundry suites, persist `suiteName`, `suiteVersion`, `generationJobId`, and `generationSource`. `suiteName` must start with a letter (`A-Z` or `a-z`); prefix derived numeric names with an alphabetic label such as `suite-`. A suite with `suiteName` still runs batch eval through `evaluation_agent_batch_eval_create`; use `evaluation_suite_get` only to resolve reviewed dataset/evaluator metadata.

# Azure Authentication Best Practices

> Source: [Microsoft — Passwordless connections for Azure services](https://learn.microsoft.com/azure/developer/intro/passwordless-overview) and [Azure Identity client libraries](https://learn.microsoft.com/dotnet/azure/sdk/authentication/).

**Table of Contents:** [Golden Rule](#golden-rule) · [Authentication by Environment](#authentication-by-environment) · [Why Not DefaultAzureCredential in Production?](#why-not-defaultazurecredential-in-production) · [Production Patterns](#production-patterns) · [Local Development Setup](#local-development-setup) · [Environment-Aware Pattern](#environment-aware-pattern) · [Security Checklist](#security-checklist) · [Further Reading](#further-reading)

## Golden Rule

Use **managed identities** and **Azure RBAC** in production. Reserve `DefaultAzureCredential` for **local development only**.

## Authentication by Environment

| Environment | Recommended Credential | Why |
|---|---|---|
| **Production (Azure-hosted)** | `ManagedIdentityCredential` (system- or user-assigned) | No secrets to manage; auto-rotated by Azure |
| **Production (on-premises)** | `ClientCertificateCredential` or `WorkloadIdentityCredential` | Deterministic; no fallback chain overhead |
| **CI/CD pipelines** | `AzurePipelinesCredential` / `WorkloadIdentityCredential` | Scoped to pipeline identity |
| **Local development** | `DefaultAzureCredential` | Chains CLI, PowerShell, and VS Code credentials for convenience |

## Why Not `DefaultAzureCredential` in Production?

1. **Unpredictable fallback chain** — walks through multiple credential types, adding latency and making failures harder to diagnose.
2. **Broad surface area** — checks environment variables, CLI tokens, and other sources that should not exist in production.
3. **Non-deterministic** — which credential actually authenticates depends on the environment, making behavior inconsistent across deployments.
4. **Performance** — each failed credential attempt adds network round-trips before falling back to the next.

## Production Patterns

### .NET

```csharp
using Azure.Identity;

var credential = Environment.GetEnvironmentVariable("AZURE_FUNCTIONS_ENVIRONMENT") == "Development"
    ? new DefaultAzureCredential()                          // local dev — uses CLI/VS credentials
    : new ManagedIdentityCredential();                      // production — deterministic, no fallback chain
// For user-assigned identity: new ManagedIdentityCredential("<client-id>")
```

### TypeScript / JavaScript

```typescript
import { DefaultAzureCredential, ManagedIdentityCredential } from "@azure/identity";

const credential = process.env.NODE_ENV === "development"
  ? new DefaultAzureCredential()                          // local dev — uses CLI/VS credentials
  : new ManagedIdentityCredential();                      // production — deterministic, no fallback chain
// For user-assigned identity: new ManagedIdentityCredential("<client-id>")
```

### Python

```python
import os
from azure.identity import DefaultAzureCredential, ManagedIdentityCredential

credential = (
    DefaultAzureCredential()                              # local dev — uses CLI/VS credentials
    if os.getenv("AZURE_FUNCTIONS_ENVIRONMENT") == "Development"
    else ManagedIdentityCredential()                      # production — deterministic, no fallback chain
)
# For user-assigned identity: ManagedIdentityCredential(client_id="<client-id>")
```

### Java

```java
import com.azure.identity.DefaultAzureCredentialBuilder;
import com.azure.identity.ManagedIdentityCredentialBuilder;

var credential = "Development".equals(System.getenv("AZURE_FUNCTIONS_ENVIRONMENT"))
    ? new DefaultAzureCredentialBuilder().build()          // local dev — uses CLI/VS credentials
    : new ManagedIdentityCredentialBuilder().build();      // production — deterministic, no fallback chain
// For user-assigned identity: new ManagedIdentityCredentialBuilder().clientId("<client-id>").build()
```

## Local Development Setup

`DefaultAzureCredential` is ideal for local dev because it automatically picks up credentials from developer tools:

1. **Azure CLI** — `az login`
2. **Azure Developer CLI** — `azd auth login`
3. **Azure PowerShell** — `Connect-AzAccount`
4. **Visual Studio / VS Code** — sign in via Azure extension

```typescript
import { DefaultAzureCredential } from "@azure/identity";

// Local development only — uses CLI/PowerShell/VS Code credentials
const credential = new DefaultAzureCredential();
```

## Environment-Aware Pattern

Detect the runtime environment and select the appropriate credential. The key principle: use `DefaultAzureCredential` only when running locally, and a specific credential in production.

> **Tip:** Azure Functions sets `AZURE_FUNCTIONS_ENVIRONMENT` to `"Development"` when running locally. For App Service or containers, use any environment variable you control (e.g. `NODE_ENV`, `ASPNETCORE_ENVIRONMENT`).

```typescript
import { DefaultAzureCredential, ManagedIdentityCredential } from "@azure/identity";

function getCredential() {
  if (process.env.NODE_ENV === "development") {
    return new DefaultAzureCredential();          // picks up az login / VS Code creds
  }
  return process.env.AZURE_CLIENT_ID
    ? new ManagedIdentityCredential(process.env.AZURE_CLIENT_ID)  // user-assigned
    : new ManagedIdentityCredential();                            // system-assigned
}
```

## Security Checklist

- [ ] Use managed identity for all Azure-hosted apps
- [ ] Never hardcode credentials, connection strings, or keys
- [ ] Apply least-privilege RBAC roles at the narrowest scope
- [ ] Use `ManagedIdentityCredential` (not `DefaultAzureCredential`) in production
- [ ] Store any required secrets in Azure Key Vault
- [ ] Rotate secrets and certificates on a schedule
- [ ] Enable Microsoft Defender for Cloud on production resources

## Further Reading

- [Passwordless connections overview](https://learn.microsoft.com/azure/developer/intro/passwordless-overview)
- [Managed identities overview](https://learn.microsoft.com/entra/identity/managed-identities-azure-resources/overview)
- [Azure RBAC overview](https://learn.microsoft.com/azure/role-based-access-control/overview)
- [.NET authentication guide](https://learn.microsoft.com/dotnet/azure/sdk/authentication/)
- [Python identity library](https://learn.microsoft.com/python/api/overview/azure/identity-readme)
- [JavaScript identity library](https://learn.microsoft.com/javascript/api/overview/azure/identity-readme)
- [Java identity library](https://learn.microsoft.com/java/api/overview/azure/identity-readme)

# Standard Agent Setup

> ⚠️ **Warning:** This page covers Foundry's **Standard Agent Setup** (capability host + bring-your-own Cosmos DB / Azure Storage / Azure AI Search). The default `azd ai agent` flow uses **Basic Agent Setup** and does **not** provision a `capabilityHosts/agents` resource — *stop reading this page* if you arrived from `azd ai agent`. See [foundry-agent/create/create-hosted.md](../foundry-agent/create/create-hosted.md) and the canonical env vars in [environment-variables.md](https://github.com/Azure/azure-dev/blob/main/cli/azd/docs/environment-variables.md).

> **MANDATORY:** Read [Standard Agent Setup docs](https://learn.microsoft.com/en-us/azure/foundry/agents/concepts/standard-agent-setup?view=foundry) before proceeding with standard setup.

## Overview

Azure AI Foundry supports two agent setup configurations:

| Setup | Capability Host | Description |
|-------|----------------|-------------|
| **Basic** | None | Default setup. All resources are Microsoft-managed. No additional connections required. |
| **Standard** | Azure AI Services | Advanced setup. Bring-your-own storage and search connections for full control over data residency and scaling. |

## Standard Setup Connections

| Connection | Service | Required | Purpose |
|------------|---------|----------|---------|
| Thread storage | Azure Cosmos DB | ✅ Yes | Store conversation threads in your own Cosmos DB instance |
| File storage | Azure Storage | ✅ Yes | Store uploaded files in your own Azure Storage account |
| Vector store | Azure AI Search | ✅ Yes | Use your own Azure AI Search instance for vector/knowledge retrieval |
| Azure AI Services | Azure AI Services | ❌ Optional | Use OpenAI models from a different AI Services resource |

> 💡 **Tip:** Standard setup is recommended for production workloads that require control over data storage, custom vector search, or integration with models from a separate AI Services resource.

## Prerequisites

Before starting deployment, confirm the following with the user:

1. **RBAC role on the resource group:** The user must have **Owner** or **User Access Administrator** role on the target resource group. The Bicep template assigns RBAC roles (Storage Blob Data Contributor, Cosmos DB Operator, AI Search roles) to the project's managed identity — this will fail without `Microsoft.Authorization/roleAssignments/write` permission.
2. **Subscription quota:** Verify the target region has available quota for AI Services. If quota is exhausted, try an alternate region (e.g., `swedencentral`, `eastus`, `westus3`).
3. **Azure Policy compliance:** Some subscriptions enforce policies (e.g., storage accounts must disable public network access). If the Bicep template fails due to policy violations, patch the template to comply (e.g., set `publicNetworkAccess: 'Disabled'` and `defaultAction: 'Deny'` on the storage account).

## Deployment

- Standard setup always creates a **new Foundry resource and a new project**. Do not ask the user for a project endpoint — one will be provisioned as part of the deployment.
- **Always use the official Bicep template:**
  [Standard Agent Setup Bicep Template](https://github.com/azure-ai-foundry/foundry-samples/blob/main/infrastructure/infrastructure-setup-bicep/43-standard-agent-setup-with-customization/main.bicep)

> ⚠️ **Warning:** Capability host provisioning is **asynchronous** and can take 10–20 minutes. After deploying the Bicep template, you **must poll** the deployment status until it succeeds. Do not assume the setup is complete immediately.

## Post-Deployment: Model & Agent

After infrastructure provisioning succeeds:

1. **Deploy a model** to the new AI Services account (e.g., `gpt-4o`). If `GlobalStandard` SKU quota is exhausted, fall back to `Standard` SKU.
2. **Create the agent** using MCP tools (`agent_update`) or the Python SDK (`client.agents.create_version`). See [SDK Operations](../foundry-agent/create/references/sdk-operations.md) for details.

## References

- [Capability Hosts — Agent Setup Types](https://learn.microsoft.com/en-us/azure/ai-foundry/agents/concepts/capability-hosts?view=foundry)
- [Standard Agent Setup](https://learn.microsoft.com/en-us/azure/foundry/agents/concepts/standard-agent-setup?view=foundry)

# Microsoft Foundry - Python SDK Guide

Python-specific implementations for working with Microsoft Foundry.

**Table of Contents:** [Prerequisites](#prerequisites) · [Model Discovery and Deployment](#model-discovery-and-deployment-mcp) · [RAG Agent with Azure AI Search](#rag-agent-with-azure-ai-search) · [Creating Agents](#creating-agents) · [Agent Evaluation](#agent-evaluation) · [Knowledge Index Operations](#knowledge-index-operations-mcp) · [Best Practices](#best-practices) · [Error Handling](#error-handling)

## Prerequisites

```bash
pip install azure-ai-projects azure-identity azure-ai-inference openai azure-ai-evaluation python-dotenv
```

### Environment Variables

```bash
PROJECT_ENDPOINT=https://<resource>.services.ai.azure.com/api/projects/<project>
MODEL_DEPLOYMENT_NAME=gpt-4o
AZURE_AI_SEARCH_CONNECTION_NAME=my-search-connection
AI_SEARCH_INDEX_NAME=my-index
AZURE_OPENAI_ENDPOINT=https://<resource>.openai.azure.com
AZURE_OPENAI_DEPLOYMENT=gpt-4o
```

## Model Discovery and Deployment (MCP)

```python
foundry_models_list()                              # All models
foundry_models_list(publisher="OpenAI")             # Filter by publisher
foundry_models_list(search_for_free_playground=True) # Free playground models

foundry_models_deploy(
    resource_group="my-rg", deployment="gpt-4o-deployment",
    model_name="gpt-4o", model_format="OpenAI",
    azure_ai_services="my-foundry-resource",
    model_version="2024-05-13", sku_capacity=10, scale_type="Standard"
)
```

## RAG Agent with Azure AI Search

> **Auth:** `DefaultAzureCredential` is for local development. See [auth-best-practices.md](../auth-best-practices.md) for production patterns.

```python
import os
from azure.ai.projects import AIProjectClient
from azure.identity import DefaultAzureCredential
from azure.ai.agents.models import (
    AzureAISearchToolDefinition, AzureAISearchToolResource,
    AISearchIndexResource, AzureAISearchQueryType,
)

project_client = AIProjectClient(
    endpoint=os.environ["FOUNDRY_PROJECT_ENDPOINT"],
    credential=DefaultAzureCredential(),
)

azs_connection = project_client.connections.get(
    os.environ["AZURE_AI_SEARCH_CONNECTION_NAME"]
)

agent = project_client.agents.create_agent(
    model=os.environ["FOUNDRY_MODEL_DEPLOYMENT_NAME"],
    name="RAGAgent",
    instructions="You are a helpful assistant. Use the knowledge base to answer. "
        "Provide citations as: `[message_idx:search_idx†source]`.",
    tools=[AzureAISearchToolDefinition(
        azure_ai_search=AzureAISearchToolResource(indexes=[
            AISearchIndexResource(
                index_connection_id=azs_connection.id,
                index_name=os.environ["AI_SEARCH_INDEX_NAME"],
                query_type=AzureAISearchQueryType.HYBRID,
            ),
        ])
    )],
)
```

### Querying a RAG Agent (Streaming)

```python
openai_client = project_client.get_openai_client()

stream = openai_client.responses.create(
    stream=True, tool_choice="required", input="Your question here",
    extra_body={"agent": {"name": agent.name, "type": "agent_reference"}},
)
for event in stream:
    if event.type == "response.output_text.delta":
        print(event.delta, end="", flush=True)
    elif event.type == "response.output_item.done":
        if event.item.type == "message" and event.item.content[-1].type == "output_text":
            for ann in event.item.content[-1].annotations:
                if ann.type == "url_citation":
                    print(f"\nCitation: {ann.url}")
```

## Creating Agents

### Basic Agent

```python
agent = project_client.agents.create_agent(
    model=os.environ["MODEL_DEPLOYMENT_NAME"],
    name="my-agent",
    instructions="You are a helpful assistant.",
)
```

### Agent with Custom Function Tools

```python
from azure.ai.agents.models import FunctionTool, ToolSet

def get_weather(location: str, unit: str = "celsius") -> str:
    """Get the current weather for a location."""
    return f"Sunny and 22°{unit[0].upper()} in {location}"

functions = FunctionTool([get_weather])
toolset = ToolSet()
toolset.add(functions)

agent = project_client.agents.create_agent(
    model=os.environ["MODEL_DEPLOYMENT_NAME"],
    name="function-agent",
    instructions="You are a helpful assistant with tool access.",
    toolset=toolset,
)
```

### Agent with Web Search

```python
from azure.ai.projects.models import (
    PromptAgentDefinition, WebSearchPreviewTool, ApproximateLocation,
)

agent = project_client.agents.create_version(
    agent_name="WebSearchAgent",
    definition=PromptAgentDefinition(
        model=os.environ["MODEL_DEPLOYMENT_NAME"],
        instructions="Search the web for current information. Provide sources.",
        tools=[
            WebSearchPreviewTool(
                user_location=ApproximateLocation(
                    country="US", city="Seattle", region="Washington"
                )
            )
        ],
    ),
)
```

> 💡 **Tip:** `WebSearchPreviewTool` requires no external resource or connection. For Bing Grounding (which requires a dedicated Bing resource and project connection), see [Bing Grounding reference](../../foundry-agent/create/references/tool-bing-grounding.md).

### Interacting with Agents

```python
from azure.ai.agents.models import ListSortOrder

thread = project_client.agents.threads.create()
project_client.agents.messages.create(thread_id=thread.id, role="user", content="Hello")

run = project_client.agents.runs.create_and_process(thread_id=thread.id, agent_id=agent.id)
if run.status == "failed":
    print(f"Run failed: {run.last_error}")

messages = project_client.agents.messages.list(thread_id=thread.id, order=ListSortOrder.ASCENDING)
for msg in messages:
    if msg.text_messages:
        print(f"{msg.role}: {msg.text_messages[-1].text.value}")

project_client.agents.delete_agent(agent.id)
```

## Agent Evaluation

### Single Response Evaluation (MCP)

```python
foundry_agents_query_and_evaluate(
    agent_id="<agent-id>", query="What's the weather?",
    endpoint="https://my-foundry.services.ai.azure.com/api/projects/my-project",
    azure_openai_endpoint="https://my-openai.openai.azure.com",
    azure_openai_deployment="gpt-4o",
    evaluators="intent_resolution,task_adherence,tool_call_accuracy"
)

foundry_agents_evaluate(
    query="What's the weather?", response="Sunny and 22°C.",
    evaluator="intent_resolution",
    azure_openai_endpoint="https://my-openai.openai.azure.com",
    azure_openai_deployment="gpt-4o"
)
```

### Batch Evaluation

```python
from azure.ai.evaluation import AIAgentConverter, IntentResolutionEvaluator, evaluate

converter = AIAgentConverter(project_client)
converter.prepare_evaluation_data(thread_ids=["t1", "t2", "t3"], filename="eval_data.jsonl")

result = evaluate(
    data="eval_data.jsonl",
    evaluators={
        "intent_resolution": IntentResolutionEvaluator(
            azure_openai_endpoint=os.environ["AZURE_OPENAI_ENDPOINT"],
            azure_openai_deployment=os.environ["AZURE_OPENAI_DEPLOYMENT"]
        ),
    },
    output_path="./eval_results"
)
print(f"Results: {result['studio_url']}")
```

> 💡 **Tip:** Continuous evaluation requires project managed identity with **Foundry User** role and Application Insights connected to the project.

## Knowledge Index Operations (MCP)

```python
foundry_knowledge_index_list(endpoint="<project-endpoint>")
foundry_knowledge_index_schema(endpoint="<project-endpoint>", index="my-index")
```

## Best Practices

1. **Never hardcode credentials** — use environment variables and `python-dotenv`
2. **Check `run.status`** and handle `HttpResponseError` exceptions
3. **Reuse `AIProjectClient`** instances — don't create new ones per request
4. **Use type hints** in custom functions for better tool integration
5. **Use context managers** for agent cleanup

## Error Handling

```python
from azure.core.exceptions import HttpResponseError

try:
    agent = project_client.agents.create_agent(
        model=os.environ["MODEL_DEPLOYMENT_NAME"],
        name="my-agent", instructions="You are helpful."
    )
except HttpResponseError as e:
    if e.status_code == 429:
        print("Rate limited — wait and retry with exponential backoff.")
    elif e.status_code == 401:
        print("Authentication failed — check credentials.")
    else:
        print(f"Error: {e.message}")
```

### Context Manager for Agent Cleanup

```python
from contextlib import contextmanager

@contextmanager
def temporary_agent(project_client, **kwargs):
    agent = project_client.agents.create_agent(**kwargs)
    try:
        yield agent
    finally:
        project_client.agents.delete_agent(agent.id)
```

---
name: microsoft-foundry:resource/create
description: |
  Create Azure AI Services multi-service resource (Foundry resource) using Azure CLI.
  USE FOR: create Foundry resource, new AI Services resource, create multi-service resource, provision Azure AI Services, AIServices kind resource, register resource provider, enable Cognitive Services, setup AI Services account, create resource group for Foundry.
  DO NOT USE FOR: creating ML workspace hubs (use microsoft-foundry:project/create), deploying models (use microsoft-foundry:models/deploy), managing permissions (use microsoft-foundry:rbac), monitoring resource usage (use microsoft-foundry:quota).
compatibility:
  required:
    - azure-cli: ">=2.0"
  optional:
    - powershell: ">=7.0"
    - azure-portal: "any"
---

# Create Foundry Resource

This sub-skill orchestrates creation of Azure AI Services multi-service resources using Azure CLI.

> **Important:** All resource creation operations are **control plane (management)** operations. Use **Azure CLI commands** as the primary method.

> **Note:** For monitoring resource usage and quotas, use the `microsoft-foundry:quota` skill.

**Table of Contents:** [Quick Reference](#quick-reference) · [When to Use](#when-to-use) · [Prerequisites](#prerequisites) · [Core Workflows](#core-workflows) · [Important Notes](#important-notes) · [Additional Resources](#additional-resources)

## Quick Reference

| Property | Value |
|----------|-------|
| **Classification** | WORKFLOW SKILL |
| **Operation Type** | Control Plane (Management) |
| **Primary Method** | Azure CLI: `az cognitiveservices account create` |
| **Resource Type** | `Microsoft.CognitiveServices/accounts` (kind: `AIServices`) |
| **Resource Kind** | `AIServices` (multi-service) |

## When to Use

Use this sub-skill when you need to:

- **Create Foundry resource** - Provision new Azure AI Services multi-service account
- **Create resource group** - Set up resource group before creating resources
- **Register resource provider** - Enable Microsoft.CognitiveServices provider
- **Manual resource creation** - CLI-based resource provisioning

**Do NOT use for:**
- Creating ML workspace hubs/projects (use `microsoft-foundry:project/create`)
- Deploying AI models (use `microsoft-foundry:models/deploy`)
- Managing RBAC permissions (use `microsoft-foundry:rbac`)
- Monitoring resource usage (use `microsoft-foundry:quota`)

## Prerequisites

- **Azure subscription** - Active subscription ([create free account](https://azure.microsoft.com/pricing/purchase-options/azure-account))
- **Azure CLI** - Version 2.0 or later installed
- **Authentication** - Run `az login` before commands
- **RBAC roles** - One of:
  - Contributor
  - Owner
  - Custom role with `Microsoft.CognitiveServices/accounts/write`
- **Resource provider** - `Microsoft.CognitiveServices` must be registered in your subscription
  - If not registered, see [Workflow #3: Register Resource Provider](#3-register-resource-provider)
  - If you lack permissions, ask a subscription Owner/Contributor to register it or grant you `/register/action` privilege

> **Need RBAC help?** See [microsoft-foundry:rbac](../../rbac/rbac.md) for permission management.

## Core Workflows

### 1. Create Resource Group

**Command Pattern:** "Create a resource group for my Foundry resources"

#### Steps

1. **Ask user preference**: Use existing or create new resource group
2. **If using existing**: List and let user select from available groups (0-4: show all, 5+: show 5 most recent with "Other" option)
3. **If creating new**: Ask user to choose region, then create

```bash
# List existing resource groups
az group list --query "[-5:].{Name:name, Location:location}" --out table

# Or create new
az group create --name <rg-name> --location <location>
az group show --name <rg-name> --query "{Name:name, Location:location, State:properties.provisioningState}"
```

See [Detailed Workflow Steps](./references/workflows.md) for complete instructions.

---

### 2. Create Foundry Resource

**Command Pattern:** "Create a new Azure AI Services resource"

#### Steps

1. **Verify prerequisites**: Check Azure CLI, authentication, and provider registration
2. **Choose location**: Always ask user to select region (don't assume resource group location)
3. **Create resource**: Use `--kind AIServices` and `--sku S0` (only supported tier)
4. **Verify and get keys**

```bash
# Create Foundry resource
az cognitiveservices account create \
  --name <resource-name> \
  --resource-group <rg> \
  --kind AIServices \
  --sku S0 \
  --location <location> \
  --yes

# Verify and get keys
az cognitiveservices account show --name <resource-name> --resource-group <rg>
az cognitiveservices account keys list --name <resource-name> --resource-group <rg>
```

**Important:** S0 (Standard) is the only supported SKU - F0 free tier not available for AIServices.

See [Detailed Workflow Steps](./references/workflows.md) for complete instructions.

---

### 3. Register Resource Provider

**Command Pattern:** "Register Cognitive Services provider"

Required when first creating Cognitive Services in subscription or if you get `ResourceProviderNotRegistered` error.

```bash
# Register provider (requires Owner/Contributor role)
az provider register --namespace Microsoft.CognitiveServices
az provider show --namespace Microsoft.CognitiveServices --query "registrationState"
```

If you lack permissions, ask a subscription Owner/Contributor to register it or use `microsoft-foundry:rbac` skill.

See [Detailed Workflow Steps](./references/workflows.md) for complete instructions.

---

## Important Notes

- **Resource kind must be `AIServices`** for multi-service Foundry resources
- **SKU must be S0** (Standard) - F0 free tier not available for AIServices
- Always ask user to choose location - different regions may have varying availability

---

## Additional Resources

- [Common Patterns](./references/patterns.md) - Quick setup patterns and command reference
- [Troubleshooting](./references/troubleshooting.md) - Common errors and solutions
- [Azure AI Services documentation](https://learn.microsoft.com/en-us/azure/ai-services/multi-service-resource?pivots=azcli)

# Common Patterns: Create Foundry Resource

**Table of Contents:** [Pattern A: Quick Setup](#pattern-a-quick-setup) · [Pattern B: Multi-Region Setup](#pattern-b-multi-region-setup) · [Quick Commands Reference](#quick-commands-reference)

## Pattern A: Quick Setup

Complete setup in one go:

```bash
# Ask user: "Use existing resource group or create new?"

# ==== If user chooses "Use existing" ====
# Count and list existing resource groups
TOTAL_RG_COUNT=$(az group list --query "length([])" -o tsv)
az group list --query "[-5:].{Name:name, Location:location}" --out table

# Based on count: show appropriate list and options
# User selects resource group
RG="<selected-rg-name>"

# Fetch details to verify
az group show --name $RG --query "{Name:name, Location:location, State:properties.provisioningState}"
# Then skip to creating Foundry resource below

# ==== If user chooses "Create new" ====
# List regions and ask user to choose
az account list-locations --query "[].{Region:name}" --out table

# Variables
RG="rg-ai-services"  # New resource group name
LOCATION="westus2"  # User's chosen location
RESOURCE_NAME="my-foundry-resource"

# Create new resource group
az group create --name $RG --location $LOCATION

# Verify creation
az group show --name $RG --query "{Name:name, Location:location, State:properties.provisioningState}"

# Create Foundry resource in user's chosen location
az cognitiveservices account create \
  --name $RESOURCE_NAME \
  --resource-group $RG \
  --kind AIServices \
  --sku S0 \
  --location $LOCATION \
  --yes

# Get endpoint and keys
echo "Resource created successfully!"
az cognitiveservices account show \
  --name $RESOURCE_NAME \
  --resource-group $RG \
  --query "{Endpoint:properties.endpoint, Location:location}"

az cognitiveservices account keys list \
  --name $RESOURCE_NAME \
  --resource-group $RG
```

## Pattern B: Multi-Region Setup

Create resources in multiple regions:

```bash
# Variables
RG="rg-ai-services"
REGIONS=("eastus" "westus2" "westeurope")

# Create resource group
az group create --name $RG --location eastus

# Create resources in each region
for REGION in "${REGIONS[@]}"; do
  RESOURCE_NAME="foundry-${REGION}"
  echo "Creating resource in $REGION..."

  az cognitiveservices account create \
    --name $RESOURCE_NAME \
    --resource-group $RG \
    --kind AIServices \
    --sku S0 \
    --location $REGION \
    --yes

  echo "Resource $RESOURCE_NAME created in $REGION"
done

# List all resources
az cognitiveservices account list --resource-group $RG --output table
```

## Quick Commands Reference

```bash
# Count total resource groups to determine which scenario applies
az group list --query "length([])" -o tsv

# Check existing resource groups (up to 5 most recent)
# 0 → create new | 1-4 → select or create | 5+ → select/other/create
az group list --query "[-5:].{Name:name, Location:location}" --out table

# If 5+ resource groups exist and user selects "Other", show all
az group list --query "[].{Name:name, Location:location}" --out table

# If user selects existing resource group, fetch details to verify and get location
az group show --name <selected-rg-name> --query "{Name:name, Location:location, State:properties.provisioningState}"

# List available regions (for creating new resource group)
az account list-locations --query "[].{Region:name}" --out table

# Create resource group (if needed)
az group create --name rg-ai-services --location westus2

# Create Foundry resource
az cognitiveservices account create \
  --name my-foundry-resource \
  --resource-group rg-ai-services \
  --kind AIServices \
  --sku S0 \
  --location westus2 \
  --yes

# List resources in group
az cognitiveservices account list --resource-group rg-ai-services

# Get resource details
az cognitiveservices account show \
  --name my-foundry-resource \
  --resource-group rg-ai-services

# Delete resource
az cognitiveservices account delete \
  --name my-foundry-resource \
  --resource-group rg-ai-services
```

# Troubleshooting: Create Foundry Resource

## Resource Creation Failures

### ResourceProviderNotRegistered

**Solution:**
1. If you have Owner/Contributor role, register the provider:
   ```bash
   az provider register --namespace Microsoft.CognitiveServices
   ```
2. If you lack permissions, ask a subscription Owner or Contributor to register it
3. Alternatively, ask them to grant you the `/register/action` privilege

### InsufficientPermissions

**Solution:**
```bash
# Check your role assignments
az role assignment list --assignee <your-user-id> --subscription <subscription-id>

# You need: Contributor, Owner, or custom role with Microsoft.CognitiveServices/accounts/write
```

Use `microsoft-foundry:rbac` skill to manage permissions.

### LocationNotAvailableForResourceType

**Solution:**
```bash
# List available regions for Cognitive Services
az provider show --namespace Microsoft.CognitiveServices \
  --query "resourceTypes[?resourceType=='accounts'].locations" --out table

# Choose different region from the list
```

### ResourceNameNotAvailable

Resource name must be globally unique. Try adding a unique suffix:

```bash
UNIQUE_SUFFIX=$(date +%s)
az cognitiveservices account create \
  --name "foundry-${UNIQUE_SUFFIX}" \
  --resource-group <rg> \
  --kind AIServices \
  --sku S0 \
  --location <location> \
  --yes
```

## Resource Shows as Failed

**Check provisioning state:**
```bash
az cognitiveservices account show \
  --name <resource-name> \
  --resource-group <rg> \
  --query "properties.provisioningState"
```

If `Failed`, delete and recreate:
```bash
# Delete failed resource
az cognitiveservices account delete \
  --name <resource-name> \
  --resource-group <rg>

# Recreate
az cognitiveservices account create \
  --name <resource-name> \
  --resource-group <rg> \
  --kind AIServices \
  --sku S0 \
  --location <location> \
  --yes
```

## Cannot Access Keys

**Error:** `AuthorizationFailed` when listing keys

**Solution:** You need `Cognitive Services User` or higher role on the resource.

Use `microsoft-foundry:rbac` skill to grant appropriate permissions.

## External Resources

- [Create multi-service resource](https://learn.microsoft.com/en-us/azure/ai-services/multi-service-resource?pivots=azcli)
- [Azure AI Services documentation](https://learn.microsoft.com/en-us/azure/ai-services/)
- [Azure regions with AI Services](https://azure.microsoft.com/global-infrastructure/services/?products=cognitive-services)

# Detailed Workflows: Create Foundry Resource

**Table of Contents:** [Workflow 1: Create Resource Group](#workflow-1-create-resource-group---detailed-steps) · [Workflow 2: Create Foundry Resource](#workflow-2-create-foundry-resource---detailed-steps) · [Workflow 3: Register Resource Provider](#workflow-3-register-resource-provider---detailed-steps)

## Workflow 1: Create Resource Group - Detailed Steps

### Step 1: Ask user preference

Ask the user which option they prefer:
1. Use an existing resource group
2. Create a new resource group

### Step 2a: If user chooses "Use existing resource group"

Count and list existing resource groups:

```bash
# Count total resource groups
TOTAL_RG_COUNT=$(az group list --query "length([])" -o tsv)

# Get list of resource groups (up to 5 most recent)
az group list --query "[-5:].{Name:name, Location:location}" --out table
```

**Handle based on count:**

**If 0 resources found:**
- Inform user: "No existing resource groups found"
- Ask if they want to create a new one, then proceed to Step 2b

**If 1-4 resources found:**
- Display all X resource groups to the user
- Let user select from the list
- Fetch the selected resource group details:
  ```bash
  az group show --name <selected-rg-name> --query "{Name:name, Location:location, State:properties.provisioningState}"
  ```
- Display details to user, then proceed to create Foundry resource

**If 5+ resources found:**
- Display the 5 most recent resource groups
- Present options:
  1. Select from the 5 displayed
  2. Other (see all resource groups)
- If user selects a resource group, fetch details:
  ```bash
  az group show --name <selected-rg-name> --query "{Name:name, Location:location, State:properties.provisioningState}"
  ```
- If user chooses "Other", show all:
  ```bash
  az group list --query "[].{Name:name, Location:location}" --out table
  ```
  Then let user select, and fetch details as above
- Display details to user, then proceed to create Foundry resource

### Step 2b: If user chooses "Create new resource group"

1. List available Azure regions:

```bash
az account list-locations --query "[].{Region:name}" --out table
```

Common regions:
- `eastus`, `eastus2` - US East Coast
- `westus`, `westus2`, `westus3` - US West Coast
- `centralus` - US Central
- `westeurope`, `northeurope` - Europe
- `southeastasia`, `eastasia` - Asia Pacific

2. Ask user to choose a region from the list above

3. Create resource group in the chosen region:

```bash
az group create \
  --name <resource-group-name> \
  --location <user-chosen-location>
```

4. Verify creation:

```bash
az group show --name <resource-group-name> --query "{Name:name, Location:location, State:properties.provisioningState}"
```

Expected output: `State: "Succeeded"`

## Workflow 2: Create Foundry Resource - Detailed Steps

### Step 1: Verify prerequisites

```bash
# Check Azure CLI version (need 2.0+)
az --version

# Verify authentication
az account show

# Check resource provider registration status
az provider show --namespace Microsoft.CognitiveServices --query "registrationState"
```

If provider not registered, see Workflow #3: Register Resource Provider.

### Step 2: Choose location

**Always ask the user to choose a location.** List available regions and let the user select:

```bash
# List available regions for Cognitive Services
az account list-locations --query "[].{Region:name, DisplayName:displayName}" --out table
```

Common regions for AI Services:
- `eastus`, `eastus2` - US East Coast
- `westus`, `westus2`, `westus3` - US West Coast
- `centralus` - US Central
- `westeurope`, `northeurope` - Europe
- `southeastasia`, `eastasia` - Asia Pacific

> **Important:** Do not automatically use the resource group's location. Always ask the user which region they prefer.

### Step 3: Create Foundry resource

```bash
az cognitiveservices account create \
  --name <resource-name> \
  --resource-group <rg> \
  --kind AIServices \
  --sku S0 \
  --location <location> \
  --yes
```

**Parameters:**
- `--name`: Unique resource name (globally unique across Azure)
- `--resource-group`: Existing resource group name
- `--kind`: **Must be `AIServices`** for multi-service resource
- `--sku`: Must be **S0** (Standard - the only supported tier for AIServices)
- `--location`: Azure region (**always ask user to choose** from available regions)
- `--yes`: Auto-accept terms without prompting

### Step 4: Verify resource creation

```bash
# Check resource details to verify creation
az cognitiveservices account show \
  --name <resource-name> \
  --resource-group <rg>

# View endpoint and configuration
az cognitiveservices account show \
  --name <resource-name> \
  --resource-group <rg> \
  --query "{Name:name, Endpoint:properties.endpoint, Location:location, Kind:kind, SKU:sku.name}"
```

Expected output:
- `provisioningState: "Succeeded"`
- Endpoint URL
- SKU: S0
- Kind: AIServices

### Step 5: Get access keys

```bash
az cognitiveservices account keys list \
  --name <resource-name> \
  --resource-group <rg>
```

This returns `key1` and `key2` for API authentication.

## Workflow 3: Register Resource Provider - Detailed Steps

### When Needed

Required when:
- First time creating Cognitive Services in subscription
- Error: `ResourceProviderNotRegistered`
- Insufficient permissions during resource creation

### Steps

**Step 1: Check registration status**

```bash
az provider show \
  --namespace Microsoft.CognitiveServices \
  --query "registrationState"
```

Possible states:
- `Registered`: Ready to use
- `NotRegistered`: Needs registration
- `Registering`: Registration in progress

**Step 2: Register provider**

```bash
az provider register --namespace Microsoft.CognitiveServices
```

**Step 3: Wait for registration**

Registration typically takes 1-2 minutes. Check status:

```bash
az provider show \
  --namespace Microsoft.CognitiveServices \
  --query "registrationState"
```

Wait until state is `Registered`.

**Step 4: Verify registration**

```bash
az provider list --query "[?namespace=='Microsoft.CognitiveServices']"
```

### Required Permissions

To register a resource provider, you need one of:
- **Subscription Owner** role
- **Contributor** role
- **Custom role** with `Microsoft.*/register/action` permission

**If you are not the subscription owner:**
1. Ask someone with the **Owner** or **Contributor** role to register the provider for you
2. Alternatively, ask them to grant you the `/register/action` privilege so you can register it yourself

**Alternative registration methods:**
- **Azure CLI** (recommended): `az provider register --namespace Microsoft.CognitiveServices`
- **Azure Portal**: Navigate to Subscriptions → Resource providers → Microsoft.CognitiveServices → Register
- **PowerShell**: `Register-AzResourceProvider -ProviderNamespace Microsoft.CognitiveServices`

---
name: private-network
description: "Answer questions about and deploy Microsoft Foundry with network isolation. Covers BYO VNet, Managed VNet, hybrid patterns, private endpoints, and Bicep deployment. WHEN: 'Foundry networking', 'BYO VNet vs managed VNet', 'deploy Foundry in private VNet', 'private endpoints for Foundry'. DO NOT USE FOR: generic Azure networking without Foundry."
license: MIT
allowed-tools: Read, Write, Bash, AskUserQuestion, microsoft_docs_search, microsoft_docs_fetch
---

# Microsoft Foundry Private Networking

## Quick Reference

| Property | Value |
|----------|-------|
| **Best for** | Foundry with VNet isolation, private endpoints, subnet delegation, APIM + Foundry, VPN/Bastion access |
| **Tools** | Azure CLI |
| **MCP Tools** | `AskUserQuestion` - ask user questions; `microsoft_docs_search` - verify facts before presenting; `microsoft_docs_fetch` - fetch full Learn pages for validation |
| **Workflow** | Ground in Learn → Gather → Plan → Scaffold → Validate → Deploy → Test |

### Key Documentation

| Topic | URL |
|-------|-----|
| Networking options (decision) | https://learn.microsoft.com/azure/foundry/agents/concepts/networking-options |
| Network isolation | https://learn.microsoft.com/azure/foundry/how-to/configure-private-link |
| Agent Service VNet | https://learn.microsoft.com/azure/foundry/agents/how-to/virtual-networks |
| Networking deep dive (subnet/IP) | https://learn.microsoft.com/azure/foundry/agents/concepts/agents-networking-deep-dive |
| Managed VNet | https://learn.microsoft.com/azure/foundry/how-to/managed-virtual-network |
| Feature limitations | https://learn.microsoft.com/azure/foundry/how-to/configure-private-link#foundry-feature-limitations |

## When to Use

- User asks about Foundry networking, private endpoints, or VNet isolation
- User asks about BYO VNet, Managed VNet, or hybrid patterns
- User wants to deploy Foundry agents in a private network
- User needs APIM integration with private Foundry agents

**Do NOT use for:**
- Public Foundry setup without VNet → use [project/create](../../project/create/create-foundry-project.md)
- Bare Foundry resource without networking → use [resource/create](../create/create-foundry-resource.md)

---

## Step 0 — Ground in Microsoft Learn
Use `microsoft_docs_fetch` to get docs from Key Documentation sources.
Use `microsoft_docs_search` to verify any technical fact before presenting it to the user. If Learn contradicts a reference file, **Learn wins**. Cite the URL. If Learn doesn't cover it, say so — do not invent facts, limits, flags, or compatibility claims.

---

## End-to-End Deployment Workflow

> **Important:** All following steps are mandatory. Communicate the plan with the user before acting.

## Step 1 — Gather Requirements

Read [references/intake.md](references/intake.md). One pass, three tiers:
- **Tier 1 (Core):** Subscription, VNet model, agents, region, RG, VNet — determine approach at the end
- **Tier 2 (Architecture):** DNS, topology, NSG, on-prem, identity, BYO resources
- **Tier 3 (Enterprise):** Model, client access, auth, policies, monitoring

Determine the approach (official template / adapt closest / extend user’s IaC) at the end of Tier 1. Continue through Tiers 2–3.

---

## Step 2 — Plan Generation

Use the confirmed requirements from [references/intake.md](references/intake.md).

**OFFICIAL path:** Load the template's README from its GitHub URL (via [references/template-index.md](references/template-index.md)). Run `microsoft_docs_search` for its prerequisites. Present a deployment plan using the user's actual values.

**ADAPT path:** Load the closest template's README. Present a deployment plan highlighting what will be modified from the base template.

**EXTEND path:** Load [references/custom-template-adaptation.md](references/custom-template-adaptation.md). Read the user's existing template. Follow the gap analysis framework to present what's covered, what's missing, and any issues. Get approval before modifying.

Get confirmation before proceeding.

---

## Step 3 — Scaffold & Parameterize

Read [references/scaffold.md](references/scaffold.md).

---

## Step 4 — Pre-Deployment Validation

Catch blockers **before** deploying. These checks apply to all paths.

**Sovereign cloud:** Run `az cloud show --query name -o tsv`. If `AzureUSGovernment` or `AzureChinaCloud`, check whether the templates being used (official or user-provided) handle sovereign cloud endpoints. Official templates hardcode `core.windows.net` and Azure Public AAD endpoints.

**RBAC:** Verify deploying identity has Owner, or Contributor + User Access Administrator.

**Policy:** Run `az deployment group what-if`. Fix any violations before deploying.

**Quota:**

```bash
az cognitiveservices account list-skus --location <region> --kind AIServices -o table
```

**Provider Registrations:** `Microsoft.CognitiveServices`, `Microsoft.DocumentDB`, `Microsoft.Search`, `Microsoft.Network`, and `Microsoft.App` (required for agent subnet delegation). The template README lists the authoritative set.

> Do NOT deploy until all pre-flight checks pass.

---

## Step 5 — Deploy & Track

**OFFICIAL / ADAPT path:** Read [references/deploy.md](references/deploy.md) for deployment command, monitoring, and error recovery.

**EXTEND path:** Deploy using the user's existing deployment workflow (their CLI commands, pipeline, or CI/CD). The monitoring and error recovery guidance in [references/deploy.md](references/deploy.md) still applies.

---

## Step 6 — Test & Validate

Read [references/post-deployment-validation.md](references/post-deployment-validation.md). These checks apply to all paths — PE verification, RBAC audit, `publicNetworkAccess` audit, and end-to-end agent test work regardless of how the infrastructure was deployed.

If any test fails, run `microsoft_docs_search` for the error before attempting remediation.

---

## Error Handling

> ⚠️ **Critical retry rule:** If a deployment fails after the capability host step starts, a service association (`legionservicelink`) stays on the agent subnet. Simplest retry: use a **new VNet name**. To reuse the same subnet, first purge the account and delete the capability host, then wait for the link to clear before redeploying. See [references/deploy.md](references/deploy.md).

For all other errors, check `microsoft_docs_search` for current remediation before acting.

# Custom Template Adaptation

For the EXTEND path — when the user has existing Bicep or Terraform templates.

## Instructions

1. **Read** the user's existing template files. Understand the resource graph: what's defined, how resources reference each other, what naming conventions are used.

2. **Analyze** the template against the user's requirements (from [intake.md](intake.md)) and the Foundry private networking documentation validated in the intake step. Identify:
   - Resources already present and correctly configured
   - Resources present but misconfigured (wrong settings, missing properties)
   - Resources missing entirely
   - Dependency or wiring issues (e.g., PEs referencing wrong subnet, DNS zones not linked)

3. **Present** findings to the user as a gap analysis table: resource, status (✅ present / ⚠️ misconfigured / ❌ missing), and what needs to change. Include any issues found.

4. **Propose** an end-to-end plan to address all gaps — ordered by dependency. Explain what will be added, what will be modified, and why. Never overwrite existing modules — add alongside and reference existing resources.

5. **Wait** for user approval before making any changes.

6. **Implement** the approved changes. After implementation, the flow continues to Step 4 (Pre-Deployment Validation) in the main workflow.

## Retry Safety

> ⚠️ If a deployment fails after the capability host step starts, Azure Container Apps leaves a `legionservicelink` service association on the agent subnet that **cannot be removed**. On retry, use a **new subnet or new VNet** — never reuse the same agent subnet.

# Deploy & Track

Applies to all private network deployments.

## Deploy

```bash
az deployment group create \
  --resource-group <rg> \
  --template-file main.bicep \
  --parameters main.bicepparam \
  --name <deployment-name>
```

> ⚠️ Capability host provisioning is **asynchronous** (10–20 min). The CLI produces no output during this phase.

## Managed VNet — Required Post-Deploy Step

After the deployment succeeds, you **must** create the managed network's outbound private endpoint rules:

- A **self-referencing private endpoint** back to the Foundry account.
- Private endpoints to its dependent data resources.

> ⚠️ **Without them, hosted agents fail with `500 (403: Public access is disabled)`** — the agent container in Microsoft's managed VNet can't reach the account privately. Prompt agents are unaffected.

The account's managed identity must hold `Azure AI Enterprise Network Connection Approver` so the rules auto-approve.

See the Managed VNet doc (Key Documentation) for current steps, or the template README for its specific script.

## Monitor Progress

Use exponential backoff — do NOT poll every 30 seconds.

| Poll | Wait |
|------|------|
| 1st | 1 min after deploy starts |
| 2nd | 3 min after 1st |
| 3rd | 5 min after 2nd |
| 4th+ | Every 5 min |

```bash
# Overall state
az deployment group show \
  --resource-group <rg> --name <deployment-name> \
  --query "{state:properties.provisioningState,error:properties.error}" -o json

# Per-resource progress
az deployment operation group list \
  --resource-group <rg> --name <deployment-name> \
  --query "[].{resource:properties.targetResource.resourceType,state:properties.provisioningState}" -o table
```

Or block with timeout:

```bash
az deployment group wait \
  --resource-group <rg> --name <deployment-name> \
  --created --timeout 1800
```

## Error Recovery

When a deployment fails, follow this workflow:

### Step 1 — Identify the error

```bash
az deployment operation group list \
  --resource-group <rg> \
  --name <deployment-name> \
  --query "[?properties.provisioningState=='Failed'].{resource:properties.targetResource.resourceType,error:properties.statusMessage}" \
  -o json
```

### Step 2 — Resolve

Use `microsoft_docs_search` with the error code or message to find current remediation. The legionservicelink retry rule is documented in the main workflow's Error Handling section.

| Error | Likely cause | Fix |
|-------|-------------|-----|
| `legionservicelink` / subnet in use | Capability host association still linked to the agent subnet | Use a new `vnetName`, or purge the account and delete the capability host, then wait for the link to clear before reusing the subnet |
| `AuthorizationFailed` on `validate/action` | Missing Contributor role | Assign Contributor + User Access Administrator to deploying identity |
| `SubnetDelegationAlreadyExists` | Agent subnet already delegated to another resource | Use a new VNet or open a support ticket to remove the delegation |
| `disableLocalAuth` policy violation | Template defaults to `false` | Set `disableLocalAuth: true` in Bicep params |
| `defaultOutboundAccess` policy violation | Subnets missing the property | Add `defaultOutboundAccess: false` to subnet properties |

### Step 3 — Present fix to user and get approval

Before re-deploying, show the user:
- What failed and why
- What file/parameter will be changed
- The new `vnetName` to use (must be different from the failed run)

### Step 4 — Re-deploy with a new deployment name

```bash
# Update main.bicepparam: change vnetName to a new unique name
az deployment group create \
  --resource-group <rg> \
  --template-file main.bicep \
  --parameters main.bicepparam \
  --name <deployment-name>-retry
```

# End-to-End Test (VNet Access Required)

Continues from [post-deployment-validation.md](post-deployment-validation.md). Steps 1–3 there must be complete first.

## 4. VNet Access Setup

> ⚠️ The remaining tests require connectivity to the VNet.

Use `AskUserQuestion`: **"Steps 1-3 are done. The remaining tests need VNet access. How do you want to proceed?"**
Options:
- `I have a Bastion VM / jump box`
- `Set up a point-to-site VPN for me` — read [vpn-dns-setup.md](vpn-dns-setup.md)
- `I have VPN / ExpressRoute already`
- `Skip testing for now`

**Bastion VM:** User has direct access to all private endpoints from the VM. Setup is complete — do NOT proceed to Step 5.

---

## 5. End-to-End Test (VPN users only)

Three phases:
1. **Network** — DNS resolution + port 443 reachability
2. **Agent Lifecycle** — Create agent, thread, run, verify, cleanup
3. **Isolation Proof** — Repeat with VPN off — expect 403

> ⚠️ Chromium browsers may bypass VPN DNS via Secure DNS (DoH). If portal shows "Error loading agents" but CLI works, disable Secure DNS.

### Requirements

```bash
pip install azure-ai-projects azure-identity azure-ai-agents
```

### Phase 1: Network Validation

Resolve DNS and test port 443 for all private endpoints. Substitute actual resource names from the deployment.

PowerShell:

```powershell
$endpoints = @(
  '<ai-account>.services.ai.azure.com',
  '<ai-account>.openai.azure.com',
  '<ai-account>.cognitiveservices.azure.com',
  '<cosmos-account>.documents.azure.com',
  '<storage-account>.blob.core.windows.net',
  '<search-service>.search.windows.net'
)
foreach ($h in $endpoints) {
  $ip = (Resolve-DnsName $h | Where-Object {$_.IPAddress}).IPAddress
  $reach = Test-NetConnection $h -Port 443 -WarningAction SilentlyContinue
  Write-Host "$h -> $ip (reachable: $($reach.TcpTestSucceeded))"
}
```

Bash:

```bash
endpoints=(
  '<ai-account>.services.ai.azure.com'
  '<ai-account>.openai.azure.com'
  '<ai-account>.cognitiveservices.azure.com'
  '<cosmos-account>.documents.azure.com'
  '<storage-account>.blob.core.windows.net'
  '<search-service>.search.windows.net'
)
for h in "${endpoints[@]}"; do
  ip=$(dig +short "$h" | tail -n1)
  nc -z -w 3 "$h" 443 >/dev/null 2>&1 && reach=yes || reach=no
  echo "$h -> $ip (reachable: $reach)"
done
```

All should resolve to private IPs and be reachable.

Report results to the user (✅/❌ per endpoint) before proceeding to Phase 2.

### Phase 2: Agent Lifecycle Test

Create agent, thread, send message, verify response, cleanup. This exercises all 4 PEs (AI Services, Cosmos DB, Storage, AI Search).

```python
from azure.identity import DefaultAzureCredential
from azure.ai.projects import AIProjectClient

endpoint = "https://<ai-account>.services.ai.azure.com/api/projects/<project-name>"
client = AIProjectClient(endpoint=endpoint, credential=DefaultAzureCredential())
agents = client.agents

agent = agents.create_agent(model="<deployment-name>", name="vnet-test", instructions="Reply with 'OK'")
thread = agents.threads.create()
agents.messages.create(thread_id=thread.id, role="user", content="test")
run = agents.runs.create_and_process(thread_id=thread.id, agent_id=agent.id)
msgs = agents.messages.list(thread_id=thread.id)
print(f"Response: {msgs.data[0].content[0].text.value}")
agents.threads.delete(thread.id)
agents.delete_agent(agent.id)
```

Report results to the user (which PEs passed, any failures) before proceeding to Phase 3.

Ask user to disconnect VPN. Repeat Phase 2 — it should fail with 403. Report whether isolation is confirmed before proceeding to cross-check.

### Requirements Cross-Check

After testing, compare each requirement gathered in [intake.md](intake.md) against the deployed state. Flag any mismatches with remediation steps.

### Cleanup (VPN users only)

Ask if user wants to delete VPN Gateway (~$140/month) and DNS Resolver (~$180/month), or keep for ongoing access.

```bash
az network vnet-gateway delete --resource-group <rg> --name vpn-gateway-<suffix> --no-wait
az network dns-resolver delete --resource-group <rg> --name dns-resolver-<suffix> --yes
az network public-ip delete --resource-group <rg> --name vpn-gateway-pip-<suffix>
```

# Intake

Collect all inputs in one pass, tiered by priority. Extract implicit answers from the user’s message before asking. Use `AskUserQuestion` for unanswered items — batch related questions.

---

## Tier 1 — Core

### 1.0 Verify Subscription

Run:

```bash
az account show --query "{Name:name, Id:id, State:state}" -o table
```

Confirm with user. Switch if needed:

```bash
az account set --subscription "<name-or-id>"
```

### 1.1 Extract Known Answers

Scan the user's message before asking:

| User Says | Inferred |
|-----------|----------|
| "my existing VNet" / "my VNet" | BYO VNet |
| "managed virtual network" | Managed VNet |
| "user-assigned identity" / "UAI" | User-assigned identity |
| "APIM" / "API Management" | Needs APIM |
| "MCP servers on the VNet" | Needs MCP subnet |
| "I have a Bicep/Terraform template" | Extend existing IaC |
| "add Foundry to my existing infra" | Extend existing IaC |

### 1.2 Architecture Questions

For unanswered items, use `AskUserQuestion`:

**VNet model:** BYO VNet or Managed VNet?

**Agents:** Agent workloads, or just models/projects?

**Region:** Which Azure region? After answer, verify capacity:

```bash
az cognitiveservices account list-skus --location <region> --kind AIServices -o table
```

If empty, warn the user and suggest alternatives.

**Resource Group:** New or existing?

**VNet:** New or existing? If new: address space (default `192.168.0.0/16`), subnet CIDRs (agent `/24`, PE `/24`).

### 1.3 Determine Approach

Based on the answers collected, select one of three paths:

```
User has existing IaC they want to extend?
├── Yes → EXTEND
│
└── No → check template-index.md
    ├── Template fits as-is → OFFICIAL
    └── Partial or no fit → ADAPT (start from closest template)
```

**OFFICIAL:** Load [template-index.md](template-index.md), fetch the best-fit README from GitHub. Present the match using the template's descriptive name.

**ADAPT:** Fetch the closest template's README. Explain what doesn't fit, present the delta, offer to adapt.

**EXTEND:** The user has existing Bicep/Terraform — no template selection needed yet. Continue to Tier 2.

Confirm the approach with the user before continuing to Tier 2.

---

## Tier 2 — Architecture

*Skip questions already answered or not applicable.*

### BYO VNet only

**Topology:** Standalone, hub-spoke, or Azure vWAN?

**On-prem connectivity:** VPN Gateway, ExpressRoute, or none?

**DNS:** Azure-provided, custom DNS resolver, or on-prem DNS forwarding?

**Address space:** Is `192.168.0.0/16` available, or use a specific range?

**NSG / Firewall:** Existing rules on the subnets?

**Deployment executor:** Where will post-deployment commands run? (VM, Bastion, VPN, Cloud Shell)

**Subscription scope:** Same subscription/tenant, cross-subscription, or cross-tenant?

**Team ownership:** Same team controls VNet, DNS, NSG, and policy? If different team, block and get pre-approval before deploying.

### Managed VNet only

**Outbound mode:** Internet outbound (default) or approved outbound only?

**MCP:** Public MCP endpoints or private MCP on VNet?

**Client access:** Where will clients connect from? (Same VNet, peered VNet, on-prem via VPN/ER, Azure-hosted service)

### Both paths

**MCP servers:** Needed on VNet?

**APIM:** Needed?

**Identity:** System-assigned (default) or user-assigned?

**BYO resources:** Reuse existing Cosmos DB / Storage / AI Search, or create new?

> If reusing, confirm all in same region as VNet.
> If reusing AI Search, it must accept Entra ID (AAD) data-plane auth — a key-only service makes agents fail with HTTP 403. Enable with `az search service update --name <search-name> --resource-group <search-rg> --auth-options aadOrApiKey --aad-auth-failure-mode http401WithBearerChallenge`.

**Key Vault / App Insights:** If user mentions existing ones, collect resource IDs. Optional.

---

## Tier 3 — Enterprise

**Agent tools:** Which tools? (AI Search, Cosmos DB, Storage, MCP, external APIs, Bing grounding, Code Interpreter)

**Model:** Name, vendor, version. Verify version format:

| Vendor | Format | Example |
|--------|--------|---------|
| OpenAI | Date | `2025-04-14` |
| Mistral AI | Integer | `1` |
| Meta | Integer | `9` |

**Client type:** SDK, web app, Teams bot, other service?

**Client network path:** Inside VNet, peered VNet, VPN/ExpressRoute?

**Authentication:** Entra ID (recommended) or API key?

> Entra ID token audience for Foundry Agents API: `https://ai.azure.com`

**GitHub access:** Can deployment environment reach `github.com`? If not, pre-stage template.

**Azure Policy:** Known policies (e.g., `disableLocalAuth`, `defaultOutboundAccess`)? If unknown, `what-if` catches them in Step 4.

**Monitoring:** Existing Log Analytics workspace, create new, or not needed?

---

## Validate Against Learn

After collecting all requirements, validate the user's configuration against current documentation. Use `microsoft_docs_fetch` on the relevant pages below, then `microsoft_docs_search` for any requirement-specific concerns not covered.

### Reference Pages

| Topic | URL |
|-------|-----|
| Networking options (decision) | https://learn.microsoft.com/azure/foundry/agents/concepts/networking-options |
| Network isolation overview | https://learn.microsoft.com/azure/foundry/how-to/configure-private-link |
| Agent Service private networking | https://learn.microsoft.com/azure/foundry/agents/how-to/virtual-networks |
| Networking deep dive (subnet/IP) | https://learn.microsoft.com/azure/foundry/agents/concepts/agents-networking-deep-dive |
| Managed VNet configuration | https://learn.microsoft.com/azure/foundry/how-to/managed-virtual-network |
| Agent Service FAQ — VNet | https://learn.microsoft.com/azure/foundry/agents/faq#virtual-networking |
| Supported regions & availability | https://learn.microsoft.com/azure/foundry/reference/region-support |
| NSP | https://learn.microsoft.com/azure/foundry/how-to/add-foundry-to-network-security-perimeter |
| Feature Limitations | https://learn.microsoft.com/en-us/azure/foundry/how-to/configure-private-link#foundry-feature-limitations |

> These URLs may change. If a fetch returns 404, use `microsoft_docs_search` to find the current page.

If a conflict is found, present:
1. The constraint and its source URL
2. Which requirement it affects
3. Options to resolve

Do NOT proceed until all conflicts are resolved or accepted.

---

## Confirmation

Present a summary of all gathered requirements. Ask: **"Confirm this is accurate before I generate a deployment plan."**

> Do NOT proceed to Plan Generation until you validated requirements against documents and the user confirms.

# Post-Deployment Validation

Run after deployment succeeds. Steps 1-3 can run from anywhere (management plane). Steps 4-5 require VNet access.

## 1. Infrastructure Verification

### 1.1 Resource State

Verify all resources are in `Succeeded` state:

```bash
az deployment operation group list \
  --resource-group <rg> --name <deployment-name> \
  --query "[].{resource:properties.targetResource.resourceType,state:properties.provisioningState}" -o table
```

### 1.2 Private Endpoint Connections

Verify all PE connections are `Approved`:

```bash
az network private-endpoint list \
  --resource-group <rg> \
  --query "[].{name:name,status:privateLinkServiceConnections[0].privateLinkServiceConnectionState.status,resource:privateLinkServiceConnections[0].groupIds[0]}" -o table
```

### 1.3 Public Network Access Audit

Verify all resources have public access disabled:

```bash
az cognitiveservices account show --name <ai-account> --resource-group <rg> \
  --query "properties.publicNetworkAccess" -o tsv

az cosmosdb show --name <cosmos-account> --resource-group <rg> \
  --query "publicNetworkAccess" -o tsv

az storage account show --name <storage-account> --resource-group <rg> \
  --query "publicNetworkAccess" -o tsv

az search service show --name <search-service> --resource-group <rg> \
  --query "publicNetworkAccess" -o tsv
```

All should return `Disabled`.

> **T10 (Private Basic):** Steps 2-5 below do not apply — T10 has no agents, no capability host, and no BYO resources. Setup is complete after Step 1.

### 1.4 Managed VNet — Outbound Private Endpoint Rules

For Managed VNet, verify the managed network's **outbound** private endpoint rules exist, especially the self-referencing private endpoint back to the account:

```bash
az rest --method get \
  --url "https://management.azure.com/subscriptions/<sub>/resourceGroups/<rg>/providers/Microsoft.CognitiveServices/accounts/<ai-account>/managedNetworks/default/outboundRules?api-version=2025-10-01-preview" \
  --query "value[].{name:name,type:properties.type,status:properties.status}" -o table
```

Expect a self-referencing private endpoint to the account plus private endpoints to its dependent data resources. A missing self-endpoint causes hosted agents to return `500 (403)`.

## 2. RBAC Role Assignment (no VNet required)

The template does not assign data-plane roles automatically.

Assign `Azure AI Developer` at the **account** scope (management-plane):

```bash
az role assignment create \
  --role "Azure AI Developer" \
  --assignee <your-object-id-or-email> \
  --scope /subscriptions/<sub>/resourceGroups/<rg>/providers/Microsoft.CognitiveServices/accounts/<ai-account-name>
```

Assign `Foundry User` at the **project** scope (data-plane — required for `agents/read`, `agents/write`):

```bash
az role assignment create \
  --role "53ca6127-db72-4b80-b1b0-d745d6d5456d" \ # Foundry User
  --assignee <your-object-id-or-email> \
  --scope /subscriptions/<sub>/resourceGroups/<rg>/providers/Microsoft.CognitiveServices/accounts/<ai-account-name>/projects/<project-name>
```

> ⚠️ RBAC propagation can take 1–5 minutes.

## 3. Deploy a Model (no VNet required)

```bash
az cognitiveservices account deployment create \
  --resource-group <rg> \
  --name <ai-account-name> \
  --deployment-name <deployment-name> \
  --model-name <modelName> \
  --model-version <modelVersion> \
  --model-format <format> \
  --sku-name GlobalStandard \
  --sku-capacity 50
```

Fall back to `Standard` SKU if `GlobalStandard` quota is exhausted.

---

## 4. VNet Access & End-to-End Test

For the remaining steps (VNet access setup, DNS resolution, agent lifecycle test, isolation proof, cleanup), read [end-to-end-test.md](end-to-end-test.md).

# Scaffold & Parameterize

Use this reference to fetch the confirmed template and wire up parameters.

## Path A — OFFICIAL / ADAPT

If the user has no GitHub access, the template must already be present in the workspace. Do NOT attempt to fetch from GitHub.

Fetch the template from the GitHub URL in [template-index.md](template-index.md). Choose **Bicep or Terraform** based on the user's preference or existing workspace files. Fetch the **entire template folder** including subdirectories. Create the files in the user's workspace (e.g., `infra/` folder).

For ADAPT: after fetching, modify the template to match the user's requirements before parameterizing.

## Path B — EXTEND

If the user has existing Bicep or Terraform templates they want to extend, load [custom-template-adaptation.md](custom-template-adaptation.md). Follow the gap analysis there: read the user's template, identify what's present, add only the missing mandatory resources.

Set parameter values using the answers collected in [intake.md](intake.md):

| Parameter | Source |
|-----------|--------|
| Location | Region (or inferred from existing VNet) |
| VNet name / resource ID | VNet answer (new or existing) |
| VNet address space | Address space from requirements (default `192.168.0.0/16`) |
| Subnet CIDRs | Subnet answers (agent `/24`, PE `/24`, MCP `/24` if needed) |
| Existing Cosmos DB / Storage / AI Search IDs | BYO resource IDs (only if reusing) |
| Isolation mode (T18 only) | Managed VNet outbound mode (`AllowOnlyApprovedOutbound` or `AllowInternetOutbound`) |
| Model name, version, format | Model selection from requirements |
| `disableLocalAuth` | Set `true` if Azure Policy requires it |

> Do NOT run `az deployment group create` yet — validate first (next step).

# Template Index — Foundry Private Network

Official templates for deploying Microsoft Foundry. Each template may be available in Bicep, Terraform, or both — use one, not both. Choose based on the user's preference or existing workspace files. Use tools to fetch Bicep and Terraform templates to understand available templates and recognize if any matches user's requirements:

**Bicep templates:** https://github.com/microsoft-foundry/foundry-samples/tree/main/infrastructure/infrastructure-setup-bicep/

**Terraform templates:** https://github.com/microsoft-foundry/foundry-samples/tree/main/infrastructure/infrastructure-setup-terraform/

**Deployment helpers:** https://github.com/microsoft-foundry/foundry-samples/tree/main/infrastructure/infrastructure-setup-bicep/deployment-tools/ — `preflight` (pre-deploy checks) and `cleanup` (correct capability-host/subnet teardown order).

Not all templates exist in both Bicep and Terraform. Some have format-specific variants (e.g., Terraform has `15a`/`15b` for new VNet vs BYO VNet; Bicep has `15a` for evaluation-only).

## How to Use

1. Fetch the **directory listing** from the relevant repo URL above — the folder names are descriptive (e.g., `15-private-network-standard-agent-setup`, `18-managed-virtual-network`)
2. Narrow to 1–2 candidates that match the user's requirements based on folder names
3. Fetch only those candidates' READMEs for full details (prerequisites, parameters, deployment instructions)

> The root README is incomplete — do not rely on it for template discovery. Use the directory listing instead.

/*
  VPN Gateway + DNS Private Resolver
  ------------------------------------
  Post-deployment add-on for private network templates (T10, T15–T19).
  Creates a P2S VPN Gateway (AAD auth, OpenVPN) and a DNS Private Resolver
  so the user can connect from their dev machine and resolve private DNS zones.

  Note: VPN Gateway deployment takes 30-45 minutes.
*/

@description('Name of the existing VNet from the Foundry deployment')
param vnetName string

@description('Resource group of the existing VNet. Defaults to the deployment resource group.')
param vnetResourceGroup string = resourceGroup().name

// ── Existing VNet ──
resource vnet 'Microsoft.Network/virtualNetworks@2024-05-01' existing = {
  name: vnetName
  scope: resourceGroup(vnetResourceGroup)
}

var location = vnet.location

@description('CIDR for GatewaySubnet — agent must compute from available VNet space')
param gatewaySubnetCidr string

@description('CIDR for DNS resolver inbound subnet — agent must compute from available VNet space')
param dnsResolverSubnetCidr string

@description('VPN client address pool — must not overlap with VNet')
param vpnClientAddressPool string = '172.16.201.0/24'

@description('Azure AD tenant ID for VPN authentication')
param aadTenantId string

@description('Unique suffix for resource naming')
param suffix string

// AAD constants for Azure Public cloud only.
// Sovereign clouds (AzureUSGovernment, AzureChinaCloud) require different audience/issuer values.
// The intake step (az cloud show) warns users before reaching this template.
var aadAudience = 'c632b3df-fb67-4d84-bdcf-b95ad541b5c8'
var aadIssuer = 'https://sts.windows.net/${aadTenantId}/'
var aadTenant = 'https://login.microsoftonline.com/${aadTenantId}/'

// ── Add subnets ──
resource gatewaySubnet 'Microsoft.Network/virtualNetworks/subnets@2024-05-01' = {
  parent: vnet
  name: 'GatewaySubnet'
  properties: {
    addressPrefix: gatewaySubnetCidr
    defaultOutboundAccess: false
  }
}

// NOTE: NRMS policy may auto-deploy an NSG on this subnet.
// Ensure the NSG allows inbound UDP/TCP port 53 (DNS) from the VPN client address pool.
resource dnsResolverSubnet 'Microsoft.Network/virtualNetworks/subnets@2024-05-01' = {
  parent: vnet
  name: 'dns-resolver-inbound'
  properties: {
    addressPrefix: dnsResolverSubnetCidr
    defaultOutboundAccess: false
    delegations: [
      {
        name: 'dns-resolver-delegation'
        properties: {
          serviceName: 'Microsoft.Network/dnsResolvers'
        }
      }
    ]
  }
  dependsOn: [gatewaySubnet] // serialize subnet updates
}

// ── Public IP for VPN Gateway ──
resource vpnGatewayPip 'Microsoft.Network/publicIPAddresses@2024-05-01' = {
  name: 'vpn-gateway-pip-${suffix}'
  location: location
  sku: {
    name: 'Standard'
  }
  zones: ['1', '2', '3']
  properties: {
    publicIPAllocationMethod: 'Static'
  }
}

// ── VPN Gateway ──
resource vpnGateway 'Microsoft.Network/virtualNetworkGateways@2024-05-01' = {
  name: 'vpn-gateway-${suffix}'
  location: location
  properties: {
    gatewayType: 'Vpn'
    vpnType: 'RouteBased'
    sku: {
      name: 'VpnGw1AZ'
      tier: 'VpnGw1AZ'
    }
    ipConfigurations: [
      {
        name: 'default'
        properties: {
          publicIPAddress: {
            id: vpnGatewayPip.id
          }
          subnet: {
            id: gatewaySubnet.id
          }
        }
      }
    ]
    vpnClientConfiguration: {
      vpnClientAddressPool: {
        addressPrefixes: [vpnClientAddressPool]
      }
      vpnClientProtocols: ['OpenVPN']
      vpnAuthenticationTypes: ['AAD']
      aadTenant: aadTenant
      aadAudience: aadAudience
      aadIssuer: aadIssuer
    }
  }
}

// ── DNS Private Resolver ──
resource dnsResolver 'Microsoft.Network/dnsResolvers@2022-07-01' = {
  name: 'dns-resolver-${suffix}'
  location: location
  properties: {
    virtualNetwork: {
      id: vnet.id
    }
  }
}

resource dnsInboundEndpoint 'Microsoft.Network/dnsResolvers/inboundEndpoints@2022-07-01' = {
  parent: dnsResolver
  name: 'inbound'
  location: location
  properties: {
    ipConfigurations: [
      {
        privateIpAllocationMethod: 'Dynamic'
        subnet: {
          id: dnsResolverSubnet.id
        }
      }
    ]
  }
}

// ── Outputs ──
output vpnGatewayName string = vpnGateway.name
output vpnGatewayId string = vpnGateway.id
output vpnPublicIpAddress string = vpnGatewayPip.properties.ipAddress
output dnsResolverInboundIp string = dnsInboundEndpoint.properties.ipConfigurations[0].privateIpAddress

# VPN Gateway & DNS Private Resolver Setup

Post-deployment add-on for private network templates (T10, T15–T19). Creates a point-to-site VPN Gateway and DNS Private Resolver so the user can connect from their dev machine and resolve private DNS zones.

## Assumptions

| Property | Value | Rationale |
|----------|-------|-----------|
| Auth | Microsoft Entra ID (AAD) only | No certificate management |
| Tunnel | OpenVPN | Cross-platform, Azure VPN Client |
| Gateway SKU | VpnGw1AZ | Zone-redundant, same cost as VpnGw1 |
| GatewaySubnet | /24 recommended | Agent computes from available VNet space |
| DNS resolver subnet | /28 minimum | Agent computes from available VNet space |
| Client address pool | `172.16.201.0/24` | Non-overlapping with VNet |

## Subnet Layout

Adds two subnets to the existing VNet. Uses the next available range after the agent and PE subnets.

| Subnet | CIDR (default) | Purpose | Delegation |
|--------|----------------|---------|------------|
| `GatewaySubnet` | Computed | VPN Gateway (name is required by Azure) | None |
| `dns-resolver-inbound` | Computed | DNS Private Resolver inbound endpoint | `Microsoft.Network/dnsResolvers` |

> ⚠️ **Warning:** `GatewaySubnet` is a reserved name — Azure requires this exact name for VPN Gateway.

## Pre-Deployment

### 1. Discover Available Subnets

List existing subnets to find free address space:

```bash
az network vnet subnet list \
  --resource-group <rg> --vnet-name <vnet-name> \
  --query "[].{name:name,cidr:addressPrefix}" -o table
```

Pick the next unused `/24` for `GatewaySubnet` and the next unused `/28` for `dns-resolver-inbound`. Both must not overlap with any existing subnet.

Example: if subnets `.0.0/24`, `.1.0/24`, `.2.0/24` are in use → use `192.168.3.0/24` for GatewaySubnet, `192.168.4.0/28` for dns-resolver-inbound.

### 2. Collect Remaining Inputs

| Parameter | Source |
|-----------|--------|
| `vnetName` | From main deployment |
| `vnetResourceGroup` | Resource group containing the VNet (omit if same as deployment RG) |
| `resourceGroupName` | Resource group for this deployment |
| `gatewaySubnetCidr` | Computed in step 1 |
| `dnsResolverSubnetCidr` | Computed in step 1 |
| `suffix` | From main deployment (or generate unique) |
| `aadTenantId` | From `az account show --query tenantId` |

### 3. Check VPN Gateway Quota

```bash
az network list-usages --location <location> \
  --query "[?name.value=='VirtualNetworkGateways'].{limit:limit,current:currentValue}" -o table
```

## Bicep Template

Template: [vpn-dns-setup.bicep](vpn-dns-setup.bicep)

| Parameter | Required | Default | Description |
|-----------|----------|---------|-------------|
| `vnetName` | Yes | — | Name of the existing VNet |
| `vnetResourceGroup` | No | Deployment RG | Resource group of the existing VNet (for BYO VNets in a different RG) |
| `aadTenantId` | Yes | — | Entra ID tenant ID for VPN auth |
| `suffix` | Yes | — | Unique suffix for resource naming |
| `gatewaySubnetCidr` | Yes | — | GatewaySubnet CIDR (computed from VNet) |
| `dnsResolverSubnetCidr` | Yes | — | DNS resolver inbound subnet CIDR (computed from VNet) |
| `vpnClientAddressPool` | No | `172.16.201.0/24` | VPN client address pool |

**Creates:** GatewaySubnet, dns-resolver-inbound subnet, Public IP (zonal), VPN Gateway (VpnGw1AZ, P2S AAD/OpenVPN), DNS Private Resolver with inbound endpoint.

## Deploy

```bash
az deployment group create \
  --resource-group <rg> \
  --template-file vpn-dns-setup.bicep \
  --parameters vnetName='<vnet-name>' aadTenantId='<tenant-id>' suffix='<suffix>' \
    gatewaySubnetCidr='<computed-cidr>' dnsResolverSubnetCidr='<computed-cidr>' \
  --name vpn-dns-setup
```

> ⚠️ **VPN Gateway provisioning takes 20–45 minutes.** This is normal. Do not cancel.

Monitor:

```bash
az deployment group show \
  --resource-group <rg> --name vpn-dns-setup \
  --query "{state:properties.provisioningState}" -o tsv
```

## Post-Deployment

### 1. Get DNS Resolver Inbound IP

```bash
az network dns-resolver inbound-endpoint show \
  --resource-group <rg> \
  --dns-resolver-name dns-resolver-<suffix> \
  --name inbound \
  --query "ipConfigurations[0].privateIpAddress" -o tsv
```

Save this IP — the VPN client needs it as custom DNS.

### 2. Connect via VPN

Provide the user with these instructions (substitute actual resource name and DNS IP):

1. Go to **Azure Portal** → `vpn-gateway-<suffix>` → **Point-to-site configuration** → **Download VPN client**
2. Extract the ZIP → edit `AzureVPN/azurevpnconfig.xml` — replace:
   ```xml
   <clientconfig i:nil="true" />
   ```
   with:
   ```xml
   <clientconfig>
     <dnsservers>
       <dnsserver><dns-resolver-inbound-ip></dnsserver>
     </dnsservers>
   </clientconfig>
   ```
3. Open [Azure VPN Client](https://aka.ms/azvpnclientdownload) → **Import** the modified `azurevpnconfig.xml` → **Connect**

Use `AskUserQuestion`: **"Let me know when you're connected so I can verify DNS resolution."**

> Do NOT proceed to verification until the user confirms they are connected.

### 3. Verify DNS Resolution

After connecting via VPN, verify private DNS zones resolve correctly:

```bash
nslookup <ai-account-name>.services.ai.azure.com
nslookup <cosmos-account>.documents.azure.com
nslookup <storage-account>.blob.core.windows.net
```

Each should resolve to a private IP (`192.168.x.x`), not a public IP.

### 4. VPN Setup Complete

DNS resolves to private IPs — VPN is working. Return to [post-deployment-validation.md](post-deployment-validation.md) **Step 5** to run the end-to-end tests.

## Troubleshooting

| Problem | Cause | Fix |
|---------|-------|-----|
| VPN connects but DNS doesn't resolve | Custom DNS not set in VPN client profile | Add DNS resolver inbound IP as custom DNS server |
| `nslookup` returns public IP | Private DNS zones not linked to VNet | Verify DNS zone VNet links: `az network private-dns zone list -g <rg>` |
| VPN client auth fails | Wrong tenant or app not consented | Verify `tenantId`, ensure Azure VPN enterprise app is consented in the tenant |
| Gateway deployment times out | Normal — VPN GW takes 20-45 min | Wait and re-check with `az deployment group show` |
| Subnet conflict | CIDR overlaps with existing subnet | Use different CIDRs for `gatewaySubnetCidr` / `dnsResolverSubnetCidr` |
| DNS resolver queries blocked | NRMS auto-deployed NSG missing DNS rules | Add inbound allow rule for UDP/TCP port 53 from VPN client address pool to the `dns-resolver-inbound` subnet NSG |