Installation
Install with CLI
Recommended
gh skills-hub install azure-diagnosticsDon't have the extension? Run gh extension install samueltauil/skills-hub first.
Download and extract to your repository:
.github/skills/azure-diagnostics/ Extract the ZIP to .github/skills/ in your repo. The folder name must match azure-diagnostics for Copilot to auto-discover it.
Skill Files (51)
SKILL.md 6.2 KB
---
name: azure-diagnostics
description: "Debug Azure production issues on Azure using AppLens, Azure Monitor, resource health, and safe triage. WHEN: debug production issues, troubleshoot app service, app service high CPU, app service deployment failure, troubleshoot container apps, troubleshoot functions, troubleshoot AKS, VM RDP, Linux SSH, VM black screen, can't connect to VM, reset VM password, NSG or firewall blocking, kubectl cannot connect, kube-system/CoreDNS failures, pod pending, crashloop, node not ready, upgrade failures, analyze logs, KQL, insights, image pull failures, cold start issues, health probe failures, resource health, root cause of errors, troubleshoot event hubs, troubleshoot service bus, messaging SDK error, AMQP connection failure, message lock lost, service bus dead letter."
license: MIT
metadata:
author: Microsoft
version: "1.2.6"
---
# Azure Diagnostics
> **AUTHORITATIVE GUIDANCE — MANDATORY COMPLIANCE**
>
> This document is the **official source** for debugging and troubleshooting Azure production issues. Follow these instructions to diagnose and resolve common Azure service problems systematically.
## Triggers
Activate this skill when user wants to:
- Debug or troubleshoot production issues
- Diagnose errors in Azure services
- Analyze application logs or metrics
- Fix image pull, cold start, or health probe issues
- Investigate why Azure resources are failing
- Find root cause of application errors
- Troubleshoot App Service issues (high CPU, deployment failures, crashes, slow responses, TLS/custom domains)
- Respond to prompts like "troubleshoot app service", "app service high CPU", or "app service deployment failure"
- Troubleshoot Azure Function Apps (invocation failures, timeouts, binding errors)
- Find the App Insights or Log Analytics workspace linked to a Function App
- Troubleshoot AKS clusters, nodes, pods, ingress, or Kubernetes networking issues
- Troubleshoot Azure VM connectivity issues (RDP/SSH failures, port 3389/22 timeouts, NSG or firewall blocking, credential resets)
- Troubleshoot Azure Messaging SDK issues (Event Hubs, Service Bus connection failures, AMQP errors, message lock issues)
## Rules
1. Start with systematic diagnosis flow
2. Use AppLens (MCP) for AI-powered diagnostics when available
3. Check resource health before deep-diving into logs
4. Select appropriate troubleshooting guide based on service type
5. Document findings and attempted remediation steps
6. Route AKS incidents to the dedicated AKS troubleshooting document
---
## Quick Diagnosis Flow
1. **Identify symptoms** - What's failing?
2. **Check resource health** - Is Azure healthy?
3. **Review logs** - What do logs show?
4. **Analyze metrics** - Performance patterns?
5. **Investigate recent changes** - What changed?
---
## Troubleshooting Guides by Service
| Service | Common Issues | Reference |
|---------|---------------|-----------|
| **Container Apps** | Image pull failures, cold starts, health probes, port mismatches | [container-apps/](references/container-apps/README.md) |
| **App Service** | High CPU, deployment failures, crashes, slow responses, TLS/custom domains | [app-service/](references/app-service/README.md) |
| **Function Apps** | App details, invocation failures, timeouts, binding errors, cold starts, missing app settings | [functions/](references/functions/README.md) |
| **AKS** | Cluster access, nodes, `kube-system`, scheduling, crash loops, ingress, DNS, upgrades | [AKS Troubleshooting](troubleshooting/aks/aks-troubleshooting.md) |
| **Compute** | VM RDP/SSH connectivity, NSG/firewall blocks, credential resets, VM agent/tooling issues | [VM Connectivity Troubleshooting](troubleshooting/compute/vm-troubleshooting.md) |
| **Messaging** | Event Hubs & Service Bus SDK errors, AMQP failures, message lock, connectivity | [Messaging Troubleshooting](troubleshooting/messaging/README.md) |
---
## Routing
- Keep Container Apps and Function Apps diagnostics in this parent skill.
- Route active AKS incidents, AKS-specific intake, evidence gathering, and remediation guidance to [AKS Troubleshooting](troubleshooting/aks/aks-troubleshooting.md).
- Route Azure VM RDP/SSH connectivity, NSG/firewall, credential reset, and VM agent troubleshooting to [VM Connectivity Troubleshooting](troubleshooting/compute/vm-troubleshooting.md).
- Route Azure Messaging SDK troubleshooting (Event Hubs, Service Bus) to [Messaging Troubleshooting](troubleshooting/messaging/README.md).
---
## Quick Reference
### Common Diagnostic Commands
```bash
# Check resource health
az resource show --ids RESOURCE_ID
# View activity log
az monitor activity-log list -g RG --max-events 20
# Container Apps logs
az containerapp logs show --name APP -g RG --follow
# Function App logs (query App Insights traces)
az monitor app-insights query --apps APP-INSIGHTS -g RG \
--analytics-query "traces | where timestamp > ago(1h) | order by timestamp desc | take 50"
```
### AppLens (MCP Tools)
For AI-powered diagnostics, use:
```
mcp_azure_mcp_applens
intent: "diagnose issues with <resource-name>"
command: "diagnose"
parameters:
resourceId: "<resource-id>"
Provides:
- Automated issue detection
- Root cause analysis
- Remediation recommendations
```
### Azure Monitor (MCP Tools)
For querying logs and metrics:
```
mcp_azure_mcp_monitor
intent: "query logs for <resource-name>"
command: "logs_query"
parameters:
workspaceId: "<workspace-id>"
query: "<KQL-query>"
```
See [kql-queries.md](references/kql-queries.md) for common diagnostic queries.
---
## Check Azure Resource Health
### Using MCP
```
mcp_azure_mcp_resourcehealth
intent: "check health status of <resource-name>"
command: "get"
parameters:
resourceId: "<resource-id>"
```
### Using CLI
```bash
# Check specific resource health
az resource show --ids RESOURCE_ID
# Check recent activity
az monitor activity-log list -g RG --max-events 20
```
---
## References
- [KQL Query Library](references/kql-queries.md)
- [Azure Resource Graph Queries](references/azure-resource-graph.md)
- [App Service Troubleshooting](references/app-service/README.md)
- [Function Apps Troubleshooting](references/functions/README.md)
- [VM Connectivity Troubleshooting](troubleshooting/compute/vm-troubleshooting.md)
- [Messaging Troubleshooting](troubleshooting/messaging/README.md)
references/app-service/
README.md 7.2 KB
# App Service Troubleshooting
## Common Issues Matrix
| Symptom | Likely Cause | Action |
|---------|--------------|-----------|
| High CPU / memory | Runaway process, inefficient code | Use Process Explorer via Kudu, scale up |
| Deployment failure | Build error, locked files, quota | Check Kudu logs at `https://APP.scm.azurewebsites.net/api/deployments` to look for details on build errors, locked files or lack of storage quota |
| App crash / restart | Unhandled exception, OOM kill | Review Event Log and STDERR in Diagnose & Solve |
| Slow responses | Downstream dependency, no caching | Enable request tracing, check dependency calls |
| 502 / 503 errors | App not starting, port conflict | Check STDERR logs, verify startup command |
| TLS / domain errors | Certificate expired, DNS mismatch | `az webapp config ssl list`, verify CNAME |
| Health check failure | Endpoint not returning 200 | Verify health check path responds within 2 min |
---
## High CPU / Memory Diagnosis
**Diagnose:**
```bash
# Check app metrics
az monitor metrics list --resource APP_RESOURCE_ID \
--metric "CpuPercentage,MemoryPercentage" --interval PT1M --output table
# View running processes via ARM Processes API (Entra ID auth)
az rest --method get \
--uri "/subscriptions/<subscription-id>/resourceGroups/<resource-group>/providers/Microsoft.Web/sites/<app-name>/processes?api-version=2024-04-01"
```
**Fix:** Scale up (`az appservice plan update -n <app-service-plan-name> -g <resource-group> --sku P1V3`) or profile the app via Kudu Process Explorer at `https://APP.scm.azurewebsites.net/ProcessExplorer/` to identify hot paths.
---
## Deployment Failure Analysis
**Diagnose:**
```bash
# List deployment history
az webapp deployment list -n APP -g RG --output table
# View deployment log for a specific deployment
az webapp log deployment show -n APP -g RG --deployment-id DEPLOY_ID
# Stream build logs from Kudu
az webapp log tail -n APP -g RG
```
**KQL — Failed deployments:**
```kql
// Replace <app-service-resource-id> with the full resource ID, for example:
// /subscriptions/<subscription-id>/resourceGroups/<resource-group>/providers/Microsoft.Web/sites/<app-name>
AppServicePlatformLogs
| where TimeGenerated > ago(24h)
| where Level == "Error" and _ResourceId == "<app-service-resource-id>"
| project TimeGenerated, Level, Message
| order by TimeGenerated desc
```
**Common deployment failures:**
| Error Message | Cause | Fix |
|---------------|-------|-----|
| `WEBSITE_RUN_FROM_PACKAGE=1` but no package | Missing zip deploy artifact | Redeploy with `az webapp deploy --src-path app.zip` |
| `Error building on server` | Oryx build failure | Check build logs, pin runtime version |
| `Locked file` during deploy | Files in use | Set an environment variable named `MSDEPLOY_RENAME_LOCKED_FILES=1` on the App Service resource to enable MSDeploy to rename locked files. |
---
## Application Crash / Restart Diagnosis
**Diagnose:**
```bash
# Check recent restarts via activity log
az monitor activity-log list -g RG --resource-id APP_RESOURCE_ID \
--max-events 10 --query "[?operationName.value=='Microsoft.Web/sites/restart/action']"
# View STDERR/STDOUT (Linux)
az webapp log download -n APP -g RG --log-file logs.zip
```
**KQL — App crashes and errors:**
```kql
AppServiceConsoleLogs
| where TimeGenerated > ago(1h)
| where ResultDescription contains "error" or ResultDescription contains "fatal"
| project TimeGenerated, ResultDescription
| order by TimeGenerated desc
| take 50
```
**Health check failures:**
```bash
# Show health check config
az webapp show -n APP -g RG --query "siteConfig.healthCheckPath"
# Test the endpoint directly
curl -s -o /dev/null -w "%{http_code}" https://APP.azurewebsites.net/health
```
> ⚠️ **Warning:** If the health check fails on >50% of instances for 1 hour, the instance is replaced.
---
## Slow Response Time Investigation
**Diagnose:**
```bash
# Check average response time
az monitor metrics list --resource APP_RESOURCE_ID \
--metric "HttpResponseTime" --interval PT5M --aggregation Average --output table
# Enable failed request tracing
az webapp log config -n APP -g RG --failed-request-tracing true
```
**KQL — Slow requests with dependency analysis:**
```kql
AppServiceHTTPLogs
| where TimeGenerated > ago(1h)
| where TimeTaken > 5000
| project TimeGenerated, CsUriStem, ScStatus, TimeTaken, CsHost
| order by TimeTaken desc
| take 20
```
**Auto-Heal — Automatic mitigation:**
```bash
# Configure auto-heal to recycle on slow requests
az webapp config set -n APP -g RG \
--auto-heal-enabled true \
--generic-configurations '{"autoHealRules":{"triggers":{"slowRequests":{"timeTaken":"00:00:30","count":10,"timeInterval":"00:02:00"}},"actions":{"actionType":"Recycle"}}}'
```
---
## Custom Domain / TLS Certificate Issues
**Diagnose:**
```bash
# List custom domains
az webapp config hostname list -g RG --webapp-name APP --output table
# List TLS certificates
az webapp config ssl list -g RG --output table
# Check SSL binding
az webapp config ssl show --certificate-name CERT -g RG
```
| Symptom | Cause | Fix |
|---------|-------|-----|
| `ERR_CERT_DATE_INVALID` | Certificate expired | If certificate came from an external certificate authority, renew with `az webapp config ssl upload` and upload a new certificate or enable managed certificates to allow Azure to provide a free TLS/SSL certificate |
| `DNS_PROBE_FINISHED_NXDOMAIN` | CNAME not configured | Add CNAME record pointing to `APP.azurewebsites.net` |
| `SSL binding not found` | Missing SNI binding | Add the missing SNI binding using `az webapp config ssl bind --certificate-thumbprint THUMB --ssl-type SNI -n APP -g RG` |
| Managed cert pending | DNS validation incomplete | Verify TXT record `asuid.DOMAIN` matches custom domain verification ID |
---
## AZ CLI or MCP Tools for App Service Diagnostics
| Tool | Command | Use When |
|----------|---------|----------|
| `Azure CLI` | `az webapp list` | List all web apps in subscription |
| `Azure CLI` | `az webapp show -n APP -g RG` | Get app config, stack, status |
| `Azure CLI` | `az webapp config appsettings list -n APP -g RG` | Check env vars and connection strings |
| `Azure CLI` | `az webapp deployment slot list -n APP -g RG` | Compare slot configurations |
| `mcp_azure_mcp_appservice` | `appservice_webapp_diagnostic_diagnose` | AI-powered root cause analysis |
| `mcp_azure_mcp_monitor` | `monitor_resource_log_query` | Run KQL against Log Analytics |
| `mcp_azure_mcp_resourcehealth` | `get` | Check platform-level health status |
> 💡 **Tip:** Start with `mcp_azure_mcp_appservice` (`diagnose`) — it automatically runs relevant detectors and surfaces the most likely root cause before you dig into logs manually.
---
## Combined Diagnostic Script
Use the [`appservice-diagnostics`](../../scripts/appservice-diagnostics.sh) script
([PowerShell](../../scripts/appservice-diagnostics.ps1)) to collect everything in one call.
It prints clearly labeled sections — app config, recent deployments, app settings, and
custom domains — and a summary line describing what it collected. Interpreting the output
remains your job.
```powershell
..\..\scripts\appservice-diagnostics.ps1 -Name <app> -ResourceGroup <rg>
```
```bash
../../scripts/appservice-diagnostics.sh --name <app> --resource-group <rg>
```
references/
azure-resource-graph.md 2.9 KB
# Azure Resource Graph Queries for Diagnostics
Azure Resource Graph (ARG) enables fast, cross-subscription resource querying using KQL via `az graph query`. Use it to check resource health, find degraded resources, and correlate incidents.
## How to Query
Use the `extension_cli_generate` MCP tool to generate `az graph query` commands:
```yaml
mcp_azure_mcp_extension_cli_generate
intent: "query Azure Resource Graph to <describe what you want to diagnose>"
cli-type: "az"
```
Or construct directly:
```bash
az graph query -q "<KQL>" --query "data[].{name:name, type:type}" -o table
```
> ⚠️ **Prerequisite:** `az extension add --name resource-graph`
## Key Tables
| Table | Contains |
|-------|----------|
| `Resources` | All ARM resources (name, type, location, properties, tags) |
| `HealthResources` | Resource health availability status |
| `ServiceHealthResources` | Azure service health events and incidents |
| `ResourceContainers` | Subscriptions, resource groups, management groups |
## Diagnostics Query Patterns
**Check resource health status across resources:**
```kql
HealthResources
| where type =~ 'microsoft.resourcehealth/availabilitystatuses'
| project name, availabilityState=properties.availabilityState, reasonType=properties.reasonType
```
**Find resources in unhealthy or degraded state:**
```kql
HealthResources
| where type =~ 'microsoft.resourcehealth/availabilitystatuses'
| where properties.availabilityState != 'Available'
| project name, state=properties.availabilityState, reason=properties.reasonType, summary=properties.summary
```
**Query active service health incidents:**
```kql
ServiceHealthResources
| where type =~ 'microsoft.resourcehealth/events'
| where properties.Status == 'Active'
| project name, title=properties.Title, impact=properties.Impact, status=properties.Status
```
**Find resources by provisioning state (failed/stuck deployments):**
```kql
Resources
| where properties.provisioningState != 'Succeeded'
| project name, type, resourceGroup, provisioningState=properties.provisioningState
```
**Find App Services in stopped or error state:**
```kql
Resources
| where type =~ 'microsoft.web/sites'
| where properties.state != 'Running'
| project name, state=properties.state, resourceGroup, location
```
**Find Container Apps with provisioning issues:**
```kql
Resources
| where type =~ 'microsoft.app/containerapps'
| where properties.provisioningState != 'Succeeded'
| project name, provisioningState=properties.provisioningState, resourceGroup
```
## Tips
- Use `=~` for case-insensitive type matching (resource types are lowercase)
- Navigate properties with `properties.fieldName`
- Use `--first N` to limit result count
- Use `--subscriptions` to scope to specific subscriptions
- Combine ARG health data with Azure Monitor metrics for full picture
- Check `HealthResources` before deep-diving into application logs
kql-queries.md 1.3 KB
# KQL Query Reference
Essential Kusto Query Language (KQL) queries for diagnosing Azure application issues.
## Prerequisites
- Application Insights or Log Analytics workspace configured
- Diagnostic settings enabled on Azure resources
---
## Recent Errors
```kql
// Recent errors
AppExceptions
| where TimeGenerated > ago(1h)
| project TimeGenerated, Message, StackTrace
| order by TimeGenerated desc
```
## Failed Requests
```kql
// Failed requests
AppRequests
| where Success == false
| where TimeGenerated > ago(1h)
| summarize count() by Name, ResultCode
| order by count_ desc
```
## Slow Requests
```kql
// Slow requests
AppRequests
| where TimeGenerated > ago(1h)
| where DurationMs > 5000
| project TimeGenerated, Name, DurationMs
| order by DurationMs desc
```
## Dependency Failures
```kql
// Dependency failures
AppDependencies
| where Success == false
| where TimeGenerated > ago(1h)
| summarize count() by Name, ResultCode, Target
```
---
## Tips
- Always include time filter: `TimeGenerated > ago(Xh)`
- Limit results with `take 50` for large datasets
- Use `summarize` to aggregate data before analyzing
## More Resources
- [KQL Quick Reference](https://learn.microsoft.com/azure/data-explorer/kql-quick-reference)
- [Application Insights Queries](https://learn.microsoft.com/azure/azure-monitor/logs/queries)
references/container-apps/
README.md 2.9 KB
# Container Apps Troubleshooting
### Common Issues Matrix
| Symptom | Likely Cause | Quick Fix |
|---------|--------------|-----------|
| Image pull failure | ACR credentials missing | `az containerapp registry set --identity system` |
| ACR build fails | ACR Tasks disabled (free sub) | Build locally with Docker |
| Cold start timeout | min-replicas=0 | `az containerapp update --min-replicas 1` |
| Port mismatch | Wrong target port | Check Dockerfile EXPOSE matches ingress |
| App keeps restarting | Health probe failing | Verify `/health` endpoint |
### Image Pull Failures
**Diagnose:**
```bash
# Check registry configuration
az containerapp show --name APP -g RG --query "properties.configuration.registries"
# Check revision status
az containerapp revision list --name APP -g RG --output table
```
**Fix:**
```bash
az containerapp registry set \
--name APP -g RG \
--server ACR.azurecr.io \
--identity system
```
### ACR Tasks Disabled (Free Subscriptions)
**Symptom:** `az acr build` fails with "ACR Tasks is not supported"
**Fix: Build locally instead:**
```bash
docker build -t ACR.azurecr.io/myapp:v1 .
az acr login --name ACR
docker push ACR.azurecr.io/myapp:v1
```
### Cold Start Issues
**Symptom:** First request very slow or times out
**Fix:**
```bash
az containerapp update --name APP -g RG --min-replicas 1
```
### Health Probe Failures
**Symptom:** Container keeps restarting
**Check:**
```bash
# View health probe config
az containerapp show --name APP -g RG --query "properties.configuration.ingress"
# Check if /health endpoint responds
curl https://APP.REGION.azurecontainerapps.io/health
```
**Fix:** Ensure app has health endpoint returning 200:
```javascript
app.get('/health', (req, res) => res.sendStatus(200));
```
### Port Mismatch
**Symptom:** App starts but returns 502/503
**Check:**
```bash
az containerapp show --name APP -g RG --query "properties.configuration.ingress.targetPort"
```
**Verify:** App must listen on this exact port. Check:
- Dockerfile `EXPOSE` statement
- `process.env.PORT` or hardcoded port in app
### View Logs
```bash
# Stream logs (wait for replicas if scale-to-zero)
az containerapp logs show --name APP -g RG --follow
# Recent logs
az containerapp logs show --name APP -g RG --tail 100
# System logs (startup issues)
az containerapp logs show --name APP -g RG --type system
```
### Get All Diagnostic Info
Use the [`containerapp-diagnostics`](../../scripts/containerapp-diagnostics.sh) script
([PowerShell](../../scripts/containerapp-diagnostics.ps1)) to collect everything in one
call. It prints clearly labeled sections — revisions, registry config, ingress config, and
recent logs — and a summary line describing what it collected. Interpreting the output
remains your job.
```powershell
..\..\scripts\containerapp-diagnostics.ps1 -Name <app> -ResourceGroup <rg>
```
```bash
../../scripts/containerapp-diagnostics.sh --name <app> --resource-group <rg>
```
references/functions/
README.md 3.5 KB
# Function Apps Troubleshooting
## Find Linked App Insights / Log Analytics
### Preferred: Use Azure Resource Graph
A single ARG query returns the App Insights name, instrumentation key, connection string, and Log Analytics workspace for a given function app:
```bash
az graph query -q "
resources | where type =~ 'microsoft.web/sites' and name == '<func-app-name>'
| project funcName=name, rg=resourceGroup
| join kind=inner (resources | where type =~ 'microsoft.insights/components' | project appiName=name, rg=resourceGroup, instrumentationKey=properties.InstrumentationKey, connectionString=properties.ConnectionString, workspaceId=properties.WorkspaceResourceId) on rg
| project funcName, appiName, instrumentationKey, connectionString, workspaceId
" -o json
```
> 💡 **Tip:** This join matches by resource group. If App Insights is in a different resource group, use the CLI fallback below.
### Fallback: CLI Commands
#### Step 1: Get the App Insights connection string from app settings
```bash
az functionapp config appsettings list \
--name <func-app-name> -g <rg-name> \
--query "[?name=='APPLICATIONINSIGHTS_CONNECTION_STRING' || name=='APPINSIGHTS_INSTRUMENTATIONKEY']"
```
#### Step 2: Find the App Insights resource by instrumentation key
```bash
az monitor app-insights component show \
--query "[?instrumentationKey=='<key>'] | [0].{name:name, rg:resourceGroup, workspaceId:workspaceResourceId}"
```
#### Step 3: Find the Log Analytics workspace
```bash
az monitor app-insights component show --app <appinsights-name> -g <rg-name> \
--query "workspaceResourceId" -o tsv
```
### Confirm logs are flowing
Query App Insights `traces` table to verify the function app is sending telemetry:
```bash
az monitor app-insights query --apps <appinsights-name> -g <rg-name> \
--analytics-query "traces | where operation_Name != '' | take 1 | project timestamp, operation_Name, message"
```
For `FunctionAppLogs` (available in Log Analytics only, not App Insights), query the workspace directly:
```bash
az monitor log-analytics query -w <workspace-guid> \
--analytics-query "FunctionAppLogs | where _ResourceId contains '<func-app-name>' | take 5 | project TimeGenerated, FunctionName, Message, Level"
```
> ⚠️ **Classic App Insights:** Some function apps use classic App Insights without a linked Log Analytics workspace (`workspaceId` is null). In this case, `FunctionAppLogs` is **not available** — use the `traces`, `requests`, and `exceptions` tables via `az monitor app-insights query` instead. As a last resort, `az webapp log tail --name <func-app-name> -g <rg-name>` can stream live logs directly.
If results are returned, logs are flowing. If empty, verify the `APPLICATIONINSIGHTS_CONNECTION_STRING` app setting matches this App Insights instance.
> ⚠️ **Always prefer querying App Insights or Log Analytics** for function app logs. `az webapp log tail` can stream live logs directly but App Insights provides richer data, historical queries, and correlation across requests.
> 💡 **Tip:** App Insights logs can be delayed by a few minutes. If you don't see recent data, wait 3-5 minutes and query again.
---
## Check Recent Deployments
Correlate issues with recent deployments by listing deployment history:
```bash
az rest --method get \
--uri "/subscriptions/<subscription-id>/resourceGroups/<rg-name>/providers/Microsoft.Web/sites/<func-app-name>/deployments?api-version=2023-12-01"
```
Compare deployment timestamps against when errors started appearing in App Insights to identify if a deployment caused the issue.
scripts/
aks-baseline.ps1 7.0 KB
<#
.SYNOPSIS
Runs the read-only AKS cluster baseline diagnostic sweep and prints a single
labeled digest instead of many raw command dumps.
.DESCRIPTION
Gathers, in order:
1. Cluster provisioning state (az aks show)
2. Node pool summary (az aks nodepool list)
3. Recent Azure activity (az monitor activity-log list)
4. Node readiness (kubectl get nodes)
5. Unhealthy pods across namespaces (kubectl get pods -A; not Ready, bad status, or restarting)
6. kube-system health (kubectl get pods -n kube-system)
7. Recent warning events (kubectl get events -A)
8. Namespace pod overview (optional) (kubectl get pods -n <namespace>)
All steps are READ-ONLY. Each step is guarded so a single failure (for example,
kubectl not authenticated) prints a note and the sweep continues.
.PARAMETER ResourceGroup
Resource group containing the AKS cluster.
.PARAMETER Cluster
AKS cluster name.
.PARAMETER Namespace
Optional namespace for an extra pod overview.
.PARAMETER Subscription
Optional Azure subscription id. Defaults to the current subscription.
.EXAMPLE
.\aks-baseline.ps1 -ResourceGroup my-rg -Cluster my-cluster
.EXAMPLE
.\aks-baseline.ps1 -ResourceGroup my-rg -Cluster my-cluster -Namespace payments
#>
param(
[Alias("g")][string]$ResourceGroup,
[Alias("n")][string]$Cluster,
[string]$Namespace,
[string]$Subscription
)
function Show-Usage($exitCode) {
Write-Host "Usage: .\aks-baseline.ps1 -ResourceGroup <resource-group> -Cluster <cluster> [-Namespace <ns>] [-Subscription <id>]"
exit $exitCode
}
if (-not $ResourceGroup) {
Write-Error "Missing required -ResourceGroup. Provide the resource group containing the AKS cluster."
Show-Usage 1
}
if (-not $Cluster) {
Write-Error "Missing required -Cluster. Provide the AKS cluster name."
Show-Usage 1
}
$azSubArgs = @()
if ($Subscription) { $azSubArgs = @("--subscription", $Subscription) }
function Write-Section($title) {
Write-Host ""
Write-Host "=============================================================="
Write-Host "== $title"
Write-Host "=============================================================="
}
function Invoke-Step($description, [scriptblock]$action) {
try {
& $action
if ($LASTEXITCODE -ne 0) {
Write-Host " [!] Could not gather: $description (command failed or unavailable)"
}
} catch {
Write-Host " [!] Could not gather: $description ($($_.Exception.Message))"
}
}
Write-Host "AKS baseline diagnostic sweep (read-only)"
Write-Host "Resource group: $ResourceGroup"
Write-Host "Cluster: $Cluster"
if ($Namespace) { Write-Host "Namespace: $Namespace" }
# 1. Cluster provisioning state ------------------------------------------------
Write-Section "1. Cluster provisioning state"
Invoke-Step "cluster provisioning state" {
az aks show -g $ResourceGroup -n $Cluster @azSubArgs `
--query "{name:name, provisioningState:provisioningState, powerState:powerState.code, k8sVersion:currentKubernetesVersion, fqdn:fqdn}" `
-o table
}
# 2. Node pool summary ---------------------------------------------------------
Write-Section "2. Node pool summary"
Invoke-Step "node pool summary" {
az aks nodepool list -g $ResourceGroup --cluster-name $Cluster @azSubArgs `
--query "[].{name:name, mode:mode, count:count, vmSize:vmSize, state:provisioningState, powerState:powerState.code, k8sVersion:orchestratorVersion}" `
-o table
}
# 3. Recent Azure activity -----------------------------------------------------
Write-Section "3. Recent Azure activity (last 20 events)"
Invoke-Step "recent activity log" {
az monitor activity-log list -g $ResourceGroup @azSubArgs `
--max-events 20 `
--query "[].{time:eventTimestamp, operation:operationName.value, status:status.value, resource:resourceId}" `
-o table
}
# 4. Node readiness ------------------------------------------------------------
Write-Section "4. Node readiness"
Invoke-Step "node readiness" { kubectl get nodes -o wide }
# 5. Unhealthy pods ------------------------------------------------------------
# Filter on the READY and STATUS columns (not just pod phase) so container-level
# failures such as CrashLoopBackOff / ImagePullBackOff — which stay in phase
# "Running" — are caught. Terminal pods (Completed/Succeeded) are excluded so
# finished jobs are not falsely flagged.
Write-Section "5. Unhealthy pods (CrashLoopBackOff, not Ready, restarting, or bad status)"
$allPods = @(kubectl get pods -A -o wide 2>$null)
if (-not $allPods) {
Write-Host " No pods reported (or cluster unreachable)."
} else {
$header = $allPods[0]
$unhealthy = foreach ($line in ($allPods | Select-Object -Skip 1)) {
$cols = $line -split '\s+'
if ($cols.Count -lt 5) { continue }
$readyParts = $cols[2] -split '/'
$status = $cols[3]
$restarts = 0; [void][int]::TryParse($cols[4], [ref]$restarts)
$terminalOk = ($status -eq 'Completed' -or $status -eq 'Succeeded')
$notReady = ($status -eq 'Running' -and $readyParts.Count -eq 2 -and $readyParts[0] -ne $readyParts[1])
$badStatus = ($status -ne 'Running' -and -not $terminalOk)
$highRestarts = ((-not $terminalOk) -and $restarts -ge 5)
if ($notReady -or $badStatus -or $highRestarts) { $line }
}
if ($unhealthy) {
$header
$unhealthy
} else {
Write-Host " All pods are Running/Succeeded and Ready with low restart counts."
}
}
# 6. kube-system health --------------------------------------------------------
Write-Section "6. kube-system health"
Invoke-Step "kube-system pods" { kubectl get pods -n kube-system -o wide }
# 7. Recent warning events -----------------------------------------------------
Write-Section "7. Recent warning events (last 40, sorted by time)"
Invoke-Step "warning events" {
$events = kubectl get events -A --field-selector="type=Warning" --sort-by=.lastTimestamp 2>$null
if ($events) { $events | Select-Object -Last 40 }
}
# 8. Namespace pod overview (optional) ----------------------------------------
if ($Namespace) {
Write-Section "8. Pods in namespace '$Namespace'"
Invoke-Step "pods in namespace $Namespace" { kubectl get pods -n $Namespace -o wide }
}
# Summary ----------------------------------------------------------------------
Write-Section "Summary"
Write-Host "Gathered the read-only AKS baseline for cluster '$Cluster' in resource group"
Write-Host "'$ResourceGroup': Azure-side cluster/node-pool state and recent activity, then"
Write-Host "Kubernetes-side node readiness, unhealthy pods, kube-system health, and recent"
Write-Host "warning events. Review the sections above for anomalies (non-Succeeded"
Write-Host "provisioning state, NotReady nodes, unhealthy or restarting pods, warning events)"
Write-Host "before deep-diving with 'kubectl describe' / 'kubectl logs' on a specific pod."
Write-Host ""
Write-Host "No changes were made to any resource."
aks-baseline.sh 6.7 KB
#!/usr/bin/env bash
# aks-baseline.sh
# Runs the read-only AKS cluster baseline diagnostic sweep and prints a single
# labeled digest instead of many raw command dumps. Gathers, in order:
# 1. Cluster provisioning state (az aks show)
# 2. Node pool summary (az aks nodepool list)
# 3. Recent Azure activity (az monitor activity-log list)
# 4. Node readiness (kubectl get nodes)
# 5. Unhealthy pods across namespaces (kubectl get pods -A; not Ready, bad status, or restarting)
# 6. kube-system health (kubectl get pods -n kube-system)
# 7. Recent warning events (kubectl get events -A)
# 8. Namespace pod overview (optional) (kubectl get pods -n <namespace>)
#
# All steps are READ-ONLY. Each step is guarded so a single failure (for example,
# kubectl not authenticated) prints a note and the sweep continues.
#
# Usage:
# ./aks-baseline.sh -g <resource-group> -n <cluster> [--namespace <ns>] [--subscription <id>]
#
# Examples:
# ./aks-baseline.sh -g my-rg -n my-cluster
# ./aks-baseline.sh -g my-rg -n my-cluster --namespace payments
set -uo pipefail
RESOURCE_GROUP=""
CLUSTER=""
NAMESPACE=""
SUBSCRIPTION=""
usage() {
echo "Usage: $0 -g <resource-group> -n <cluster> [--namespace <ns>] [--subscription <id>]" >&2
exit "${1:-1}"
}
require_value() {
# require_value <option-name> <remaining-arg-count>
if [ "$2" -lt 2 ]; then
echo "Missing value for $1" >&2
usage 1
fi
}
while [ $# -gt 0 ]; do
case "$1" in
-g|--resource-group) require_value "$1" "$#"; RESOURCE_GROUP="$2"; shift 2 ;;
-n|--cluster) require_value "$1" "$#"; CLUSTER="$2"; shift 2 ;;
--namespace) require_value "$1" "$#"; NAMESPACE="$2"; shift 2 ;;
--subscription) require_value "$1" "$#"; SUBSCRIPTION="$2"; shift 2 ;;
-h|--help) usage 0 ;;
*) echo "Unknown argument: $1" >&2; usage 1 ;;
esac
done
[ -z "$RESOURCE_GROUP" ] && { echo "Missing required -g/--resource-group" >&2; usage 1; }
[ -z "$CLUSTER" ] && { echo "Missing required -n/--cluster" >&2; usage 1; }
AZ_SUB_ARGS=()
[ -n "$SUBSCRIPTION" ] && AZ_SUB_ARGS=(--subscription "$SUBSCRIPTION")
section() {
echo ""
echo "=============================================================="
echo "== $1"
echo "=============================================================="
}
run() {
# run "<description>" <command...>
local desc="$1"; shift
if ! "$@"; then
echo " [!] Could not gather: $desc (command failed or unavailable)"
fi
}
echo "AKS baseline diagnostic sweep (read-only)"
echo "Resource group: $RESOURCE_GROUP"
echo "Cluster: $CLUSTER"
[ -n "$NAMESPACE" ] && echo "Namespace: $NAMESPACE"
# 1. Cluster provisioning state ------------------------------------------------
section "1. Cluster provisioning state"
run "cluster provisioning state" \
az aks show -g "$RESOURCE_GROUP" -n "$CLUSTER" ${AZ_SUB_ARGS[@]+"${AZ_SUB_ARGS[@]}"} \
--query "{name:name, provisioningState:provisioningState, powerState:powerState.code, k8sVersion:currentKubernetesVersion, fqdn:fqdn}" \
-o table
# 2. Node pool summary ---------------------------------------------------------
section "2. Node pool summary"
run "node pool summary" \
az aks nodepool list -g "$RESOURCE_GROUP" --cluster-name "$CLUSTER" ${AZ_SUB_ARGS[@]+"${AZ_SUB_ARGS[@]}"} \
--query "[].{name:name, mode:mode, count:count, vmSize:vmSize, state:provisioningState, powerState:powerState.code, k8sVersion:orchestratorVersion}" \
-o table
# 3. Recent Azure activity -----------------------------------------------------
section "3. Recent Azure activity (last 20 events)"
run "recent activity log" \
az monitor activity-log list -g "$RESOURCE_GROUP" ${AZ_SUB_ARGS[@]+"${AZ_SUB_ARGS[@]}"} \
--max-events 20 \
--query "[].{time:eventTimestamp, operation:operationName.value, status:status.value, resource:resourceId}" \
-o table
# 4. Node readiness ------------------------------------------------------------
section "4. Node readiness"
run "node readiness" kubectl get nodes -o wide
# 5. Unhealthy pods ------------------------------------------------------------
# Filter on the READY and STATUS columns (not just pod phase) so container-level
# failures such as CrashLoopBackOff / ImagePullBackOff — which stay in phase
# "Running" — are caught. Terminal pods (Completed/Succeeded) are excluded so
# finished jobs are not falsely flagged.
section "5. Unhealthy pods (CrashLoopBackOff, not Ready, restarting, or bad status)"
ALL_PODS="$(kubectl get pods -A -o wide 2>/dev/null)"
if [ -z "$ALL_PODS" ]; then
echo " No pods reported (or cluster unreachable)."
else
UNHEALTHY="$(printf '%s\n' "$ALL_PODS" | awk 'NR>1 {
split($3, ready, "/");
status = $4;
restarts = $5 + 0;
terminalOk = (status == "Completed" || status == "Succeeded");
notReady = (status == "Running" && ready[1] != ready[2]);
badStatus = (status != "Running" && !terminalOk);
highRestarts = (!terminalOk && restarts >= 5);
if (notReady || badStatus || highRestarts) print
}')"
if [ -n "$UNHEALTHY" ]; then
printf '%s\n' "$ALL_PODS" | head -n 1
printf '%s\n' "$UNHEALTHY"
else
echo " All pods are Running/Succeeded and Ready with low restart counts."
fi
fi
# 6. kube-system health --------------------------------------------------------
section "6. kube-system health"
run "kube-system pods" kubectl get pods -n kube-system -o wide
# 7. Recent warning events -----------------------------------------------------
section "7. Recent warning events (last 40, sorted by time)"
run "warning events" bash -c \
"set -o pipefail; kubectl get events -A --field-selector=type=Warning --sort-by=.lastTimestamp 2>/dev/null | tail -n 40"
# 8. Namespace pod overview (optional) ----------------------------------------
if [ -n "$NAMESPACE" ]; then
section "8. Pods in namespace '$NAMESPACE'"
run "pods in namespace $NAMESPACE" kubectl get pods -n "$NAMESPACE" -o wide
fi
# Summary ----------------------------------------------------------------------
section "Summary"
echo "Gathered the read-only AKS baseline for cluster '$CLUSTER' in resource group"
echo "'$RESOURCE_GROUP': Azure-side cluster/node-pool state and recent activity, then"
echo "Kubernetes-side node readiness, unhealthy pods, kube-system health, and recent"
echo "warning events. Review the sections above for anomalies (non-Succeeded"
echo "provisioning state, NotReady nodes, unhealthy or restarting pods, warning events)"
echo "before deep-diving with 'kubectl describe' / 'kubectl logs' on a specific pod."
echo ""
echo "No changes were made to any resource."
appservice-diagnostics.ps1 2.3 KB
<#
.SYNOPSIS
Collects diagnostic information for an Azure App Service web app in one pass.
.DESCRIPTION
Prints clearly labeled sections for the given web app: app config, recent
deployments, app settings, and custom domains. The script only gathers and
labels output; it does not interpret the results.
.PARAMETER Name
Name of the App Service web app.
.PARAMETER ResourceGroup
Resource group that contains the web app.
.PARAMETER Subscription
Azure subscription ID. Defaults to the current subscription.
.EXAMPLE
.\appservice-diagnostics.ps1 -Name my-app -ResourceGroup my-rg
# Collects config, recent deployments, app settings, and custom domains for my-app
#>
param(
[string]$Name,
[string]$ResourceGroup,
[string]$Subscription
)
$ErrorActionPreference = "Continue"
if (-not $Name -or -not $ResourceGroup) {
Write-Error "Usage: .\appservice-diagnostics.ps1 -Name <app> -ResourceGroup <rg> [-Subscription <id>]"
exit 1
}
$subArgs = @()
if ($Subscription) { $subArgs = @("--subscription", $Subscription) }
Write-Host "=== App Service Diagnostics: $Name (resource group: $ResourceGroup) ==="
Write-Host "Collecting app config, recent deployments, app settings, and custom domains."
Write-Host ""
Write-Host "--- App Config ---"
az webapp show -n $Name -g $ResourceGroup @subArgs --query "{state:state, runtime:siteConfig.linuxFxVersion, healthCheck:siteConfig.healthCheckPath, alwaysOn:siteConfig.alwaysOn}" -o table
if ($LASTEXITCODE -ne 0) { Write-Host "(failed to read app config)" }
Write-Host ""
Write-Host "--- Recent Deployments (last 3) ---"
az webapp deployment list -n $Name -g $ResourceGroup @subArgs --query "[:3].{id:id, status:status, time:end_time}" -o table
if ($LASTEXITCODE -ne 0) { Write-Host "(failed to list deployments)" }
Write-Host ""
Write-Host "--- App Settings (names only) ---"
az webapp config appsettings list -n $Name -g $ResourceGroup @subArgs --query "[].name" -o tsv
if ($LASTEXITCODE -ne 0) { Write-Host "(failed to list app settings)" }
Write-Host ""
Write-Host "--- Custom Domains ---"
az webapp config hostname list -g $ResourceGroup --webapp-name $Name @subArgs -o table
if ($LASTEXITCODE -ne 0) { Write-Host "(failed to list custom domains)" }
Write-Host ""
Write-Host "=== Diagnostics collection complete for $Name ==="
appservice-diagnostics.sh 2.9 KB
#!/usr/bin/env bash
# appservice-diagnostics.sh
# Collects diagnostic information for an Azure App Service web app in one pass
# and prints it as clearly labeled sections: app config, recent deployments,
# app settings, and custom domains. The script only gathers and labels output;
# it does not interpret the results.
#
# Usage:
# ./appservice-diagnostics.sh --name <app> --resource-group <rg> [--subscription <id>]
# ./appservice-diagnostics.sh <app> <rg> [subscription-id]
#
# Examples:
# ./appservice-diagnostics.sh --name my-app --resource-group my-rg
# ./appservice-diagnostics.sh my-app my-rg
set -euo pipefail
APP=""
RG=""
SUBSCRIPTION=""
usage() {
echo "Usage: $0 --name <app> --resource-group <rg> [--subscription <id>]" >&2
}
# Requires a value to follow the given flag; errors out otherwise.
require_value() {
if [ "$2" -lt 2 ]; then
echo "Error: option '$1' requires a value." >&2
usage
exit 1
fi
}
# Support both --flag and positional styles.
POSITIONAL=()
while [ $# -gt 0 ]; do
case "$1" in
--name|-n) require_value "$1" "$#"; APP="$2"; shift 2 ;;
--resource-group|-g) require_value "$1" "$#"; RG="$2"; shift 2 ;;
--subscription|-s) require_value "$1" "$#"; SUBSCRIPTION="$2"; shift 2 ;;
--*|-?) echo "Error: unknown option '$1'." >&2; usage; exit 1 ;;
*) POSITIONAL+=("$1"); shift ;;
esac
done
if [ -z "$APP" ] && [ "${#POSITIONAL[@]}" -ge 1 ]; then APP="${POSITIONAL[0]}"; fi
if [ -z "$RG" ] && [ "${#POSITIONAL[@]}" -ge 2 ]; then RG="${POSITIONAL[1]}"; fi
if [ -z "$SUBSCRIPTION" ] && [ "${#POSITIONAL[@]}" -ge 3 ]; then SUBSCRIPTION="${POSITIONAL[2]}"; fi
if [ -z "$APP" ] || [ -z "$RG" ]; then
usage
exit 1
fi
SUB_ARGS=()
if [ -n "$SUBSCRIPTION" ]; then SUB_ARGS=(--subscription "$SUBSCRIPTION"); fi
echo "=== App Service Diagnostics: $APP (resource group: $RG) ==="
echo "Collecting app config, recent deployments, app settings, and custom domains."
echo ""
echo "--- App Config ---"
az webapp show -n "$APP" -g "$RG" "${SUB_ARGS[@]}" \
--query "{state:state, runtime:siteConfig.linuxFxVersion, healthCheck:siteConfig.healthCheckPath, alwaysOn:siteConfig.alwaysOn}" \
-o table || echo "(failed to read app config)"
echo ""
echo "--- Recent Deployments (last 3) ---"
az webapp deployment list -n "$APP" -g "$RG" "${SUB_ARGS[@]}" \
--query "[:3].{id:id, status:status, time:end_time}" -o table || echo "(failed to list deployments)"
echo ""
echo "--- App Settings (names only) ---"
az webapp config appsettings list -n "$APP" -g "$RG" "${SUB_ARGS[@]}" \
--query "[].name" -o tsv || echo "(failed to list app settings)"
echo ""
echo "--- Custom Domains ---"
az webapp config hostname list -g "$RG" --webapp-name "$APP" "${SUB_ARGS[@]}" -o table || echo "(failed to list custom domains)"
echo ""
echo "=== Diagnostics collection complete for $APP ==="
containerapp-diagnostics.ps1 2.1 KB
<#
.SYNOPSIS
Collects diagnostic information for an Azure Container App in one pass.
.DESCRIPTION
Prints clearly labeled sections for the given Container App: revisions,
registry configuration, ingress configuration, and recent logs. The script
only gathers and labels output; it does not interpret the results.
.PARAMETER Name
Name of the Container App.
.PARAMETER ResourceGroup
Resource group that contains the Container App.
.PARAMETER Subscription
Azure subscription ID. Defaults to the current subscription.
.EXAMPLE
.\containerapp-diagnostics.ps1 -Name my-app -ResourceGroup my-rg
# Collects revisions, registry/ingress config, and recent logs for my-app
#>
param(
[string]$Name,
[string]$ResourceGroup,
[string]$Subscription
)
$ErrorActionPreference = "Continue"
if (-not $Name -or -not $ResourceGroup) {
Write-Error "Usage: .\containerapp-diagnostics.ps1 -Name <app> -ResourceGroup <rg> [-Subscription <id>]"
exit 1
}
$subArgs = @()
if ($Subscription) { $subArgs = @("--subscription", $Subscription) }
Write-Host "=== Container App Diagnostics: $Name (resource group: $ResourceGroup) ==="
Write-Host "Collecting revisions, registry/ingress configuration, and recent logs."
Write-Host ""
Write-Host "--- Revisions ---"
az containerapp revision list --name $Name -g $ResourceGroup @subArgs -o table
if ($LASTEXITCODE -ne 0) { Write-Host "(failed to list revisions)" }
Write-Host ""
Write-Host "--- Registry Config ---"
az containerapp show --name $Name -g $ResourceGroup @subArgs --query "properties.configuration.registries"
if ($LASTEXITCODE -ne 0) { Write-Host "(failed to read registry config)" }
Write-Host ""
Write-Host "--- Ingress Config ---"
az containerapp show --name $Name -g $ResourceGroup @subArgs --query "properties.configuration.ingress"
if ($LASTEXITCODE -ne 0) { Write-Host "(failed to read ingress config)" }
Write-Host ""
Write-Host "--- Recent Logs (last 20 lines) ---"
az containerapp logs show --name $Name -g $ResourceGroup @subArgs --tail 20
if ($LASTEXITCODE -ne 0) { Write-Host "(failed to read logs)" }
Write-Host ""
Write-Host "=== Diagnostics collection complete for $Name ==="
containerapp-diagnostics.sh 2.7 KB
#!/usr/bin/env bash
# containerapp-diagnostics.sh
# Collects diagnostic information for an Azure Container App in one pass and
# prints it as clearly labeled sections: revisions, registry config, ingress
# config, and recent logs. The script only gathers and labels output; it does
# not interpret the results.
#
# Usage:
# ./containerapp-diagnostics.sh --name <app> --resource-group <rg> [--subscription <id>]
# ./containerapp-diagnostics.sh <app> <rg> [subscription-id]
#
# Examples:
# ./containerapp-diagnostics.sh --name my-app --resource-group my-rg
# ./containerapp-diagnostics.sh my-app my-rg
set -euo pipefail
APP=""
RG=""
SUBSCRIPTION=""
usage() {
echo "Usage: $0 --name <app> --resource-group <rg> [--subscription <id>]" >&2
}
# Requires a value to follow the given flag; errors out otherwise.
require_value() {
if [ "$2" -lt 2 ]; then
echo "Error: option '$1' requires a value." >&2
usage
exit 1
fi
}
# Support both --flag and positional styles.
POSITIONAL=()
while [ $# -gt 0 ]; do
case "$1" in
--name|-n) require_value "$1" "$#"; APP="$2"; shift 2 ;;
--resource-group|-g) require_value "$1" "$#"; RG="$2"; shift 2 ;;
--subscription|-s) require_value "$1" "$#"; SUBSCRIPTION="$2"; shift 2 ;;
--*|-?) echo "Error: unknown option '$1'." >&2; usage; exit 1 ;;
*) POSITIONAL+=("$1"); shift ;;
esac
done
if [ -z "$APP" ] && [ "${#POSITIONAL[@]}" -ge 1 ]; then APP="${POSITIONAL[0]}"; fi
if [ -z "$RG" ] && [ "${#POSITIONAL[@]}" -ge 2 ]; then RG="${POSITIONAL[1]}"; fi
if [ -z "$SUBSCRIPTION" ] && [ "${#POSITIONAL[@]}" -ge 3 ]; then SUBSCRIPTION="${POSITIONAL[2]}"; fi
if [ -z "$APP" ] || [ -z "$RG" ]; then
usage
exit 1
fi
SUB_ARGS=()
if [ -n "$SUBSCRIPTION" ]; then SUB_ARGS=(--subscription "$SUBSCRIPTION"); fi
echo "=== Container App Diagnostics: $APP (resource group: $RG) ==="
echo "Collecting revisions, registry/ingress configuration, and recent logs."
echo ""
echo "--- Revisions ---"
az containerapp revision list --name "$APP" -g "$RG" "${SUB_ARGS[@]}" -o table || echo "(failed to list revisions)"
echo ""
echo "--- Registry Config ---"
az containerapp show --name "$APP" -g "$RG" "${SUB_ARGS[@]}" \
--query "properties.configuration.registries" || echo "(failed to read registry config)"
echo ""
echo "--- Ingress Config ---"
az containerapp show --name "$APP" -g "$RG" "${SUB_ARGS[@]}" \
--query "properties.configuration.ingress" || echo "(failed to read ingress config)"
echo ""
echo "--- Recent Logs (last 20 lines) ---"
az containerapp logs show --name "$APP" -g "$RG" "${SUB_ARGS[@]}" --tail 20 || echo "(failed to read logs)"
echo ""
echo "=== Diagnostics collection complete for $APP ==="
pod-evidence.ps1 6.6 KB
<#
.SYNOPSIS
Collects the invariant, read-only AKS pod-failure evidence bundle and prints a
single labeled digest.
.DESCRIPTION
Gathers the same evidence bundle for one or more pods regardless of the symptom
(CrashLoopBackOff, OOMKilled, Pending, probe failures, ImagePullBackOff). For each
pod it collects and summarizes:
STATUS - READY / phase / restart count (kubectl get pod -o wide)
STATE - exit code, reason, last-state snippet (jsonpath over containerStatuses)
EVENTS - the Events section (kubectl describe pod)
LOGS - current container logs (tailed) (kubectl logs)
PREV LOGS - previous/crashed container logs (tailed)(kubectl logs --previous)
RESOURCES - requests/limits vs live usage (jsonpath + kubectl top pod)
This script only GATHERS and DIGESTS evidence. It never mutates cluster state.
Interpreting the digest to pick a fix stays with the caller.
.PARAMETER Pod
Pod name (single-pod mode). Omit when using -AllFailing.
.PARAMETER Namespace
Namespace of the pod. Required in single-pod mode. In -AllFailing mode, limits the
scan to this namespace.
.PARAMETER AllFailing
Auto-select every pod not in Running/Succeeded phase (across all namespaces unless
-Namespace is given) and digest each.
.PARAMETER Tail
Number of log lines to show per stream. Default 50.
.EXAMPLE
.\pod-evidence.ps1 my-api-7d9f-abcde -Namespace prod
.EXAMPLE
.\pod-evidence.ps1 -AllFailing
.EXAMPLE
.\pod-evidence.ps1 -AllFailing -Namespace prod -Tail 100
#>
param(
[Parameter(Position = 0)][string]$Pod,
[Alias("n")][string]$Namespace,
[switch]$AllFailing,
[int]$Tail = 50
)
# Best-effort: individual kubectl reads may fail (unreachable cluster, missing
# metrics-server, no previous logs). PowerShell's default $ErrorActionPreference is
# "Continue", so a single failed read is suppressed via 2>$null and the digest proceeds
# instead of aborting — no explicit assignment needed.
if (-not (Get-Command kubectl -ErrorAction SilentlyContinue)) {
Write-Error "kubectl not found on PATH."
exit 1
}
if ($Tail -le 0) {
Write-Error "-Tail must be a positive integer (got '$Tail')."
exit 2
}
function Digest-Pod {
param([string]$Ns, [string]$Name)
Write-Host "=================================================================="
Write-Host "POD: $Name NAMESPACE: $Ns"
Write-Host "=================================================================="
Write-Host "--- STATUS (ready / phase / restarts) ---"
$status = kubectl get pod $Name -n $Ns -o wide 2>&1
if ($LASTEXITCODE -eq 0 -and $status) { $status | ForEach-Object { Write-Host "$_" } } else { Write-Host "(unable to get pod)" }
Write-Host ""
Write-Host "--- STATE (exit code / reason / last state) ---"
$jp = '{range .status.containerStatuses[*]}container={.name}{"`n"} ready={.ready} restarts={.restartCount}{"`n"} current: waiting={.state.waiting.reason} running={.state.running.startedAt} terminated={.state.terminated.reason}(exit={.state.terminated.exitCode}){"`n"} lastState: terminated={.lastState.terminated.reason}(exit={.lastState.terminated.exitCode}) at {.lastState.terminated.finishedAt}{"`n"}{end}'
$state = kubectl get pod $Name -n $Ns -o jsonpath=$jp 2>$null
if ($state) { Write-Host $state } else { Write-Host "(no container status available)" }
Write-Host ""
Write-Host "--- EVENTS ---"
$desc = kubectl describe pod $Name -n $Ns 2>$null
if ($desc) {
$idx = ($desc | Select-String -Pattern '^Events:' | Select-Object -First 1).LineNumber
if ($idx) {
$desc | Select-Object -Skip ($idx - 1) | Select-Object -First 25 | ForEach-Object { Write-Host "$_" }
} else {
Write-Host "(no Events section)"
}
} else {
Write-Host "(unable to describe pod)"
}
Write-Host ""
Write-Host "--- LOGS (current, last $Tail lines) ---"
$logs = kubectl logs $Name -n $Ns --tail=$Tail 2>&1
if ($logs) { $logs | ForEach-Object { Write-Host "$_" } } else { Write-Host "(no current logs)" }
Write-Host ""
Write-Host "--- PREV LOGS (previous instance, last $Tail lines) ---"
$prev = kubectl logs $Name -n $Ns --previous --tail=$Tail 2>$null
if ($LASTEXITCODE -eq 0 -and $prev) {
Write-Host $prev
} else {
Write-Host "(no previous-instance logs - pod has not restarted or they were rotated)"
}
Write-Host ""
Write-Host "--- RESOURCES (requests/limits vs live usage) ---"
Write-Host "requests/limits:"
$res = kubectl get pod $Name -n $Ns -o jsonpath='{range .spec.containers[*]} {.name}: requests={.resources.requests} limits={.resources.limits}{"`n"}{end}' 2>$null
if ($res) { Write-Host $res } else { Write-Host " (unable to read resources)" }
Write-Host "live usage:"
$top = kubectl top pod $Name -n $Ns 2>&1
if ($top) { $top | ForEach-Object { Write-Host " $_" } } else { Write-Host " (metrics-server unavailable)" }
Write-Host ""
}
if ($AllFailing) {
$scope = if ($Namespace) { " in namespace '$Namespace'" } else { "" }
Write-Host "pod-evidence: scanning for pods not in Running/Succeeded$scope..."
$cols = "custom-columns=NS:.metadata.namespace,NAME:.metadata.name"
if ($Namespace) {
$rows = kubectl get pods -n $Namespace --field-selector=status.phase!=Running,status.phase!=Succeeded --no-headers -o $cols 2>$null
} else {
$rows = kubectl get pods -A --field-selector=status.phase!=Running,status.phase!=Succeeded --no-headers -o $cols 2>$null
}
$rows = @($rows | Where-Object { $_ -and $_.Trim() })
if ($rows.Count -eq 0) {
Write-Host "No unhealthy pods found (all pods are Running or Succeeded)."
exit 0
}
Write-Host "Found $($rows.Count) unhealthy pod(s). Collecting evidence for each below."
Write-Host ""
foreach ($row in $rows) {
$parts = ($row -split '\s+') | Where-Object { $_ }
Digest-Pod -Ns $parts[0] -Name $parts[1]
}
Write-Host "pod-evidence: done. Reviewed $($rows.Count) failing pod(s) - use the STATE/EVENTS/LOGS above to pick a fix."
} else {
if (-not $Pod) {
Write-Error "A pod name is required (or use -AllFailing)."
exit 2
}
if (-not $Namespace) {
Write-Error "-Namespace is required in single-pod mode."
exit 2
}
Write-Host "pod-evidence: collecting the read-only evidence bundle for pod '$Pod' in namespace '$Namespace'."
Write-Host ""
Digest-Pod -Ns $Namespace -Name $Pod
Write-Host "pod-evidence: done. Use the STATE/EVENTS/LOGS/RESOURCES digest above to pick a fix."
}
pod-evidence.sh 7.1 KB
#!/usr/bin/env bash
# pod-evidence.sh
# Collects the invariant, read-only AKS pod-failure evidence bundle for one or more
# pods and prints a single labeled digest. Works identically regardless of the pod
# symptom (CrashLoopBackOff, OOMKilled, Pending, probe failures, ImagePullBackOff).
#
# For each pod it gathers and summarizes:
# STATUS - READY / phase / restart count (kubectl get pod -o wide)
# STATE - exit code, reason, last-state snippet (jsonpath over containerStatuses)
# EVENTS - the Events section (kubectl describe pod)
# LOGS - current container logs (tailed) (kubectl logs)
# PREV LOGS - previous/crashed container logs (tailed)(kubectl logs --previous)
# RESOURCES - requests/limits vs live usage (jsonpath + kubectl top pod)
#
# This script only GATHERS and DIGESTS evidence. It never mutates cluster state.
# Interpreting the digest to pick a fix (exit-code / event / probe decision tables)
# stays with the caller.
#
# Usage:
# ./pod-evidence.sh <pod> -n <namespace> [--tail <n>]
# ./pod-evidence.sh --all-failing [-n <namespace>] [--tail <n>]
#
# Options:
# -n, --namespace <ns> Namespace of the pod. Required in single-pod mode.
# In --all-failing mode, limits the scan to this namespace.
# --all-failing Auto-select every pod not in Running/Succeeded phase
# (across all namespaces unless -n is given) and digest each.
# --tail <n> Number of log lines to show per stream (default 50).
# -h, --help Show this help.
#
# Examples:
# ./pod-evidence.sh my-api-7d9f-abcde -n prod
# ./pod-evidence.sh --all-failing
# ./pod-evidence.sh --all-failing -n prod --tail 100
set -euo pipefail
POD=""
NAMESPACE=""
ALL_FAILING=false
TAIL=50
usage() {
sed -n '2,40p' "$0" | sed 's/^# \{0,1\}//'
}
while [ $# -gt 0 ]; do
case "$1" in
-n|--namespace) NAMESPACE="${2:?--namespace requires a value}"; shift 2 ;;
--all-failing) ALL_FAILING=true; shift ;;
--tail) TAIL="${2:?--tail requires a value}"; shift 2 ;;
-h|--help) usage; exit 0 ;;
-*) echo "Unknown option: $1" >&2; usage; exit 2 ;;
*) POD="$1"; shift ;;
esac
done
if ! command -v kubectl >/dev/null 2>&1; then
echo "ERROR: kubectl not found on PATH." >&2
exit 1
fi
case "$TAIL" in
''|*[!0-9]*) echo "ERROR: --tail must be a positive integer (got '$TAIL')." >&2; exit 2 ;;
0) echo "ERROR: --tail must be a positive integer (got '$TAIL')." >&2; exit 2 ;;
esac
# Digest a single pod. Args: <namespace> <pod>
digest_pod() {
local ns="$1" pod="$2"
echo "=================================================================="
echo "POD: $pod NAMESPACE: $ns"
echo "=================================================================="
echo "--- STATUS (ready / phase / restarts) ---"
kubectl get pod "$pod" -n "$ns" -o wide 2>&1 || echo "(unable to get pod)"
echo ""
echo "--- STATE (exit code / reason / last state) ---"
kubectl get pod "$pod" -n "$ns" -o jsonpath='{range .status.containerStatuses[*]}container={.name}{"\n"} ready={.ready} restarts={.restartCount}{"\n"} current: waiting={.state.waiting.reason} running={.state.running.startedAt} terminated={.state.terminated.reason}(exit={.state.terminated.exitCode}){"\n"} lastState: terminated={.lastState.terminated.reason}(exit={.lastState.terminated.exitCode}) at {.lastState.terminated.finishedAt}{"\n"}{end}' 2>/dev/null \
|| echo "(no container status available)"
echo ""
echo "--- EVENTS ---"
if kubectl describe pod "$pod" -n "$ns" >/dev/null 2>&1; then
# Read the whole describe output in a single awk pass (no `head` in the pipe:
# an early-exiting `head` would SIGPIPE the upstream kubectl and, under
# `set -o pipefail`, abort the script). awk consumes all input and prints only
# the first 25 lines of the Events section.
kubectl describe pod "$pod" -n "$ns" 2>/dev/null | awk '/^Events:/{f=1} f && n<25 {print; n++}'
else
echo "(unable to describe pod)"
fi
echo ""
echo "--- LOGS (current, last $TAIL lines) ---"
kubectl logs "$pod" -n "$ns" --tail="$TAIL" 2>&1 || echo "(no current logs)"
echo ""
echo "--- PREV LOGS (previous instance, last $TAIL lines) ---"
if kubectl logs "$pod" -n "$ns" --previous --tail="$TAIL" 2>/dev/null; then
:
else
echo "(no previous-instance logs - pod has not restarted or they were rotated)"
fi
echo ""
echo "--- RESOURCES (requests/limits vs live usage) ---"
echo "requests/limits:"
kubectl get pod "$pod" -n "$ns" -o jsonpath='{range .spec.containers[*]} {.name}: requests={.resources.requests} limits={.resources.limits}{"\n"}{end}' 2>/dev/null \
|| echo " (unable to read resources)"
echo "live usage:"
kubectl top pod "$pod" -n "$ns" 2>&1 | sed 's/^/ /' || echo " (metrics-server unavailable)"
echo ""
}
if [ "$ALL_FAILING" = true ]; then
echo "pod-evidence: scanning for pods not in Running/Succeeded${NAMESPACE:+ in namespace '$NAMESPACE'}..."
# Portable, set -e-safe row collection: capture output + exit status via command
# substitution (not process substitution, whose failures don't propagate under set -e)
# so a failed scan is reported as an error instead of a misleading "no unhealthy pods".
# Avoids `mapfile`, which is unavailable in Bash 3.2 / macOS.
if [ -n "$NAMESPACE" ]; then
SCAN_ARGS=(-n "$NAMESPACE")
else
SCAN_ARGS=(-A)
fi
set +e
SCAN_OUT=$(kubectl get pods "${SCAN_ARGS[@]}" --field-selector=status.phase!=Running,status.phase!=Succeeded --no-headers -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name 2>/dev/null)
SCAN_RC=$?
set -e
if [ "$SCAN_RC" -ne 0 ]; then
echo "ERROR: unable to list pods (kubectl exited $SCAN_RC). Check your cluster context and credentials." >&2
exit 1
fi
ROWS=()
while IFS= read -r line; do
[ -n "$line" ] && ROWS+=("$line")
done <<< "$SCAN_OUT"
if [ "${#ROWS[@]}" -eq 0 ]; then
echo "No unhealthy pods found (all pods are Running or Succeeded)."
exit 0
fi
echo "Found ${#ROWS[@]} unhealthy pod(s). Collecting evidence for each below."
echo ""
for row in "${ROWS[@]}"; do
[ -z "$row" ] && continue
ns="${row%% *}"
name="${row##* }"
digest_pod "$ns" "$name"
done
echo "pod-evidence: done. Reviewed ${#ROWS[@]} failing pod(s) - use the STATE/EVENTS/LOGS above to pick a fix."
else
if [ -z "$POD" ]; then
echo "ERROR: a pod name is required (or use --all-failing)." >&2
usage
exit 2
fi
if [ -z "$NAMESPACE" ]; then
echo "ERROR: -n/--namespace is required in single-pod mode." >&2
exit 2
fi
echo "pod-evidence: collecting the read-only evidence bundle for pod '$POD' in namespace '$NAMESPACE'."
echo ""
digest_pod "$NAMESPACE" "$POD"
echo "pod-evidence: done. Use the STATE/EVENTS/LOGS/RESOURCES digest above to pick a fix."
fi
run-ig.ps1 5.6 KB
<#
.SYNOPSIS
Runs an Inspektor Gadget (IG) trace on an AKS node via `kubectl debug`.
.DESCRIPTION
Handles the mechanical, error-prone assembly of the IG invocation:
- resolves the target node from a pod (or takes a node directly)
- injects the pinned IG image + version
- applies the correct default -Timeout for the gadget type
- adds the k8s namespace/pod/container filters
- handles the special `tcpdump` gadget (pcap-ng output piped to tcpdump)
The privileged debug pod requires explicit user approval and appropriate RBAC.
Use -DryRun to print the assembled command without running it.
.PARAMETER Gadget
Gadget to run, e.g. trace_dns, snapshot_socket, tcpdump (required).
.PARAMETER Pod
Pod name; the node is resolved automatically.
.PARAMETER Namespace
Namespace of the pod (required with -Pod).
.PARAMETER Node
Run directly against a node (node-wide scope).
.PARAMETER Container
Scope to a specific container.
.PARAMETER Timeout
Override the gadget-type default timeout (seconds).
.PARAMETER Filter
Extra IG flags, passed through verbatim (e.g. -Filter --max-entries,20).
.PARAMETER Pf
tcpdump packet filter (tcpdump gadget only, e.g. "port 80").
.PARAMETER IgVersion
Override the pinned IG image tag.
.PARAMETER DryRun
Print the assembled command; do not execute.
.EXAMPLE
./run-ig.ps1 -Gadget trace_dns -Pod web-0 -Namespace default
.EXAMPLE
./run-ig.ps1 -Gadget snapshot_process -Node aks-nodepool1-1234
.EXAMPLE
./run-ig.ps1 -Gadget tcpdump -Pod web-0 -Namespace default -Pf "port 80"
.EXAMPLE
./run-ig.ps1 -Gadget traceloop -Pod web-0 -Namespace default -Filter --syscall-filters,open,connect
.EXAMPLE
./run-ig.ps1 -Gadget trace_dns -Pod web-0 -Namespace default -DryRun
#>
[CmdletBinding()]
param(
[string]$Gadget,
[string]$Pod,
[Alias('Ns')]
[string]$Namespace,
[string]$Node,
[string]$Container,
[int]$Timeout,
[string[]]$Filter,
[string]$Pf,
# Pinned IG image tag. Bump this default (and run-ig.sh) to update the IG version.
[string]$IgVersion = 'v0.51.0',
[switch]$DryRun
)
$IgImageRepo = 'mcr.microsoft.com/oss/v2/inspektor-gadget/ig'
if (-not $Gadget) {
Write-Error 'Provide -Gadget <name> (e.g. trace_dns, snapshot_socket, tcpdump).'
exit 2
}
if (-not $Node -and -not $Pod) {
Write-Error 'Provide either -Node <node> or -Pod <pod> -Namespace <namespace>.'
exit 2
}
if ($Pod -and -not $Namespace) {
Write-Error '-Pod requires -Namespace <namespace>.'
exit 2
}
if ($Pf -and $Gadget -ne 'tcpdump') {
Write-Error '-Pf is only valid for the tcpdump gadget.'
exit 2
}
# Default timeout by gadget type, inferred from the gadget name prefix.
# snapshot_* / top_* -> 5s (point-in-time / quick aggregate)
# trace_* / profile_* / tcpdump -> 30s (streaming / sampling)
function Get-DefaultTimeout([string]$g) {
switch -Wildcard ($g) {
'snapshot_*' { return 5 }
'top_*' { return 5 }
'trace_*' { return 30 }
'profile_*' { return 30 }
'tcpdump' { return 30 }
default { return 30 } # unknown gadget: use the safer streaming default
}
}
if (-not $PSBoundParameters.ContainsKey('Timeout') -or $Timeout -le 0) {
$Timeout = Get-DefaultTimeout $Gadget
}
# Resolve the node name from the pod when not given directly.
if (-not $Node) {
$Node = ((& kubectl get pod $Pod -n $Namespace -o "jsonpath={.spec.nodeName}" 2>$null) | Out-String).Trim()
if (-not $Node) {
Write-Error "Could not resolve node for pod '$Pod' in namespace '$Namespace'."
exit 1
}
}
$IgImage = "${IgImageRepo}:${IgVersion}"
# Assemble the k8s scoping filters.
$filters = @()
if ($Namespace) { $filters += @('--k8s-namespace', $Namespace) }
if ($Pod) { $filters += @('--k8s-podname', $Pod) }
if ($Container) { $filters += @('--k8s-containername', $Container) }
# Base kubectl debug invocation.
$debug = @('debug', '--profile=sysadmin', "node/$Node", '--attach', '--quiet', "--image=$IgImage", '--')
if ($Gadget -eq 'tcpdump') {
# tcpdump emits raw pcap-ng; pipe through tcpdump for readable output when available.
$igCmd = @('ig', 'run', "tcpdump:$IgVersion", '-o', 'pcap-ng') + $filters + @('--timeout', "$Timeout")
if ($Pf) { $igCmd += @('--pf', $Pf) }
if ($Filter) { $igCmd += $Filter }
}
else {
$igCmd = @('ig', 'run', "${Gadget}:$IgVersion", '-o', 'json') + $filters + @('--timeout', "$Timeout")
if ($Filter) { $igCmd += $Filter }
}
$fullArgs = $debug + $igCmd
# Pretty-print a shell-quoted version of the command for display.
function Format-Cmd([string[]]$parts) {
($parts | ForEach-Object {
if ($_ -match '\s') { '"' + $_ + '"' } else { $_ }
}) -join ' '
}
$displayCmd = 'kubectl ' + (Format-Cmd $fullArgs)
# The tcpdump gadget is only piped through `tcpdump` when that binary is present.
# Reflect the real behavior in the displayed command so -DryRun does not mislead.
$tcpdumpAvail = $Gadget -eq 'tcpdump' -and [bool](Get-Command tcpdump -ErrorAction SilentlyContinue)
if ($tcpdumpAvail) {
$displayCmd = "$displayCmd | tcpdump -nvr -"
}
Write-Host "Gadget: $Gadget"
Write-Host "Node: $Node"
Write-Host "Timeout: ${Timeout}s"
Write-Host "Image: $IgImage"
Write-Host "Command: $displayCmd"
if ($Gadget -eq 'tcpdump' -and -not $tcpdumpAvail) {
Write-Host 'Note: tcpdump not found; emitting raw pcap-ng to stdout.'
}
if ($DryRun) {
Write-Host '(dry-run: command not executed)'
exit 0
}
Write-Host "Ran gadget $Gadget on node $Node (timeout ${Timeout}s)"
if ($tcpdumpAvail) {
& kubectl @fullArgs | & tcpdump -nvr -
}
else {
& kubectl @fullArgs
}
run-ig.sh 6.5 KB
#!/usr/bin/env bash
# run-ig.sh
# Runs an Inspektor Gadget (IG) trace on an AKS node via `kubectl debug`.
#
# Handles the mechanical, error-prone assembly of the IG invocation:
# - resolves the target node from a pod (or takes a node directly)
# - injects the pinned IG image + version
# - applies the correct default --timeout for the gadget type
# - adds the k8s namespace/pod/container filters
# - handles the special `tcpdump` gadget (pcap-ng output piped to tcpdump)
#
# The privileged debug pod requires explicit user approval and appropriate RBAC.
# Use --dry-run to print the assembled command without running it.
#
# Usage:
# ./run-ig.sh --gadget <name> (--pod <pod> --ns <namespace> | --node <node>) [options]
#
# Options:
# --gadget <name> Gadget to run, e.g. trace_dns, snapshot_socket, tcpdump (required)
# --pod <pod> Pod name; the node is resolved automatically
# --ns <namespace> Namespace of the pod (required with --pod)
# --node <node> Run directly against a node (node-wide scope)
# --container <name> Scope to a specific container
# --timeout <seconds> Override the gadget-type default timeout
# --filter <arg> Extra IG flag, repeatable (e.g. --filter --max-entries --filter 20)
# --pf "<expr>" tcpdump packet filter (tcpdump gadget only, e.g. "port 80")
# --ig-version <tag> Override the pinned IG image tag (default below)
# --dry-run Print the assembled command; do not execute
#
# Examples:
# ./run-ig.sh --gadget trace_dns --pod web-0 --ns default
# ./run-ig.sh --gadget snapshot_process --node aks-nodepool1-1234
# ./run-ig.sh --gadget tcpdump --pod web-0 --ns default --pf "port 80"
# ./run-ig.sh --gadget traceloop --pod web-0 --ns default --filter --syscall-filters --filter open,connect
# ./run-ig.sh --gadget trace_dns --pod web-0 --ns default --dry-run
set -euo pipefail
# Pinned IG image tag. Bump this line (and run-ig.ps1) to update the IG version.
IG_VERSION="v0.51.0"
IG_IMAGE_REPO="mcr.microsoft.com/oss/v2/inspektor-gadget/ig"
GADGET=""
POD=""
NS=""
NODE=""
CONTAINER=""
TIMEOUT=""
PF=""
DRY_RUN="false"
EXTRA_FILTERS=()
usage() {
# Print the leading comment block (from line 2) as help, stopping at the
# first non-comment line so script code is never echoed.
awk 'NR>1 && /^#/ { sub(/^# ?/, ""); print; next } NR>1 { exit }' "$0"
}
while [[ $# -gt 0 ]]; do
case "$1" in
--gadget) GADGET="${2:?--gadget requires a value}"; shift 2;;
--pod) POD="${2:?--pod requires a value}"; shift 2;;
--ns|--namespace) NS="${2:?--ns requires a value}"; shift 2;;
--node) NODE="${2:?--node requires a value}"; shift 2;;
--container) CONTAINER="${2:?--container requires a value}"; shift 2;;
--timeout) TIMEOUT="${2:?--timeout requires a value}"; shift 2;;
--filter) EXTRA_FILTERS+=("${2:?--filter requires a value}"); shift 2;;
--pf) PF="${2:?--pf requires a value}"; shift 2;;
--ig-version) IG_VERSION="${2:?--ig-version requires a value}"; shift 2;;
--dry-run) DRY_RUN="true"; shift;;
-h|--help) usage; exit 0;;
*) echo "Unknown argument: $1" >&2; usage >&2; exit 2;;
esac
done
if [[ -z "$GADGET" ]]; then
echo "Error: --gadget is required." >&2
exit 2
fi
if [[ -z "$NODE" && -z "$POD" ]]; then
echo "Error: provide either --node <node> or --pod <pod> --ns <namespace>." >&2
exit 2
fi
if [[ -n "$POD" && -z "$NS" ]]; then
echo "Error: --pod requires --ns <namespace>." >&2
exit 2
fi
# Default timeout by gadget type, inferred from the gadget name prefix.
# snapshot_* / top_* -> 5s (point-in-time / quick aggregate)
# trace_* / profile_* / tcpdump -> 30s (streaming / sampling)
default_timeout() {
case "$1" in
snapshot_*|top_*) echo 5;;
trace_*|profile_*|tcpdump) echo 30;;
*) echo 30;; # unknown gadget: use the safer streaming default
esac
}
if [[ -z "$TIMEOUT" ]]; then
TIMEOUT="$(default_timeout "$GADGET")"
fi
# Resolve the node name from the pod when not given directly.
if [[ -z "$NODE" ]]; then
NODE="$(kubectl get pod "$POD" -n "$NS" -o jsonpath='{.spec.nodeName}')"
if [[ -z "$NODE" ]]; then
echo "Error: could not resolve node for pod '$POD' in namespace '$NS'." >&2
exit 1
fi
fi
IG_IMAGE="${IG_IMAGE_REPO}:${IG_VERSION}"
# Assemble the k8s scoping filters.
FILTERS=()
[[ -n "$NS" ]] && FILTERS+=(--k8s-namespace "$NS")
[[ -n "$POD" ]] && FILTERS+=(--k8s-podname "$POD")
[[ -n "$CONTAINER" ]] && FILTERS+=(--k8s-containername "$CONTAINER")
# Base kubectl debug invocation.
DEBUG=(kubectl debug --profile=sysadmin "node/${NODE}" --attach --quiet --image="$IG_IMAGE" --)
if [[ "$GADGET" == "tcpdump" ]]; then
# tcpdump emits raw pcap-ng; pipe through tcpdump for readable output when available.
IG_CMD=(ig run "tcpdump:${IG_VERSION}" -o pcap-ng "${FILTERS[@]}" --timeout "$TIMEOUT")
[[ -n "$PF" ]] && IG_CMD+=(--pf "$PF")
[[ ${#EXTRA_FILTERS[@]} -gt 0 ]] && IG_CMD+=("${EXTRA_FILTERS[@]}")
else
if [[ -n "$PF" ]]; then
echo "Error: --pf is only valid for the tcpdump gadget." >&2
exit 2
fi
IG_CMD=(ig run "${GADGET}:${IG_VERSION}" -o json "${FILTERS[@]}" --timeout "$TIMEOUT")
[[ ${#EXTRA_FILTERS[@]} -gt 0 ]] && IG_CMD+=("${EXTRA_FILTERS[@]}")
fi
FULL_CMD=("${DEBUG[@]}" "${IG_CMD[@]}")
# Pretty-print a shell-quoted version of the command for display.
quote_cmd() {
local out=""
local a
for a in "$@"; do
if [[ "$a" =~ [[:space:]] ]]; then
out+="\"$a\" "
else
out+="$a "
fi
done
echo "${out% }"
}
DISPLAY_CMD="$(quote_cmd "${FULL_CMD[@]}")"
# The tcpdump gadget is only piped through `tcpdump` when that binary is present.
# Reflect the real behavior in the displayed command so --dry-run does not mislead.
TCPDUMP_AVAIL="false"
if [[ "$GADGET" == "tcpdump" ]] && command -v tcpdump >/dev/null 2>&1; then
TCPDUMP_AVAIL="true"
DISPLAY_CMD="$DISPLAY_CMD | tcpdump -nvr -"
fi
echo "Gadget: $GADGET" >&2
echo "Node: $NODE" >&2
echo "Timeout: ${TIMEOUT}s" >&2
echo "Image: $IG_IMAGE" >&2
echo "Command: $DISPLAY_CMD" >&2
if [[ "$GADGET" == "tcpdump" && "$TCPDUMP_AVAIL" == "false" ]]; then
echo "Note: tcpdump not found; emitting raw pcap-ng to stdout." >&2
fi
if [[ "$DRY_RUN" == "true" ]]; then
echo "(dry-run: command not executed)" >&2
exit 0
fi
echo "Ran gadget $GADGET on node $NODE (timeout ${TIMEOUT}s)" >&2
if [[ "$TCPDUMP_AVAIL" == "true" ]]; then
"${FULL_CMD[@]}" | tcpdump -nvr -
else
"${FULL_CMD[@]}"
fi
test-messaging-connectivity.ps1 6.0 KB
<#
.SYNOPSIS
Probes reachability of an Azure Service Bus / Event Hubs namespace and prints
a normalized per-check report.
.DESCRIPTION
Runs the mechanical connectivity probe for a messaging namespace: DNS
resolution, HTTPS reachability, and TCP connectivity to the well-known
messaging ports (AMQP 5671/5672, HTTPS 443, and — with -Kafka — Event Hubs
Kafka 9093). Emits one normalized table plus a summary so the result is clear
without re-inspecting raw Test-NetConnection / Resolve-DnsName output.
A blocked port is a valid diagnostic result, not a failure. Choosing which
namespace to test and diagnosing a blocked port (IP firewall vs. corporate
proxy vs. NSG) require judgment and stay in the skill prose.
.PARAMETER Namespace
The messaging namespace. May be a full FQDN (e.g.
"contoso.servicebus.windows.net") or a bare namespace name (e.g. "contoso");
when no dot is present, ".servicebus.windows.net" is appended automatically.
.PARAMETER Kafka
Also probe the Event Hubs Kafka endpoint on port 9093.
.EXAMPLE
.\test-messaging-connectivity.ps1 -Namespace contoso
# Tests contoso.servicebus.windows.net (DNS, HTTPS, AMQP 5671/5672, 443)
.EXAMPLE
.\test-messaging-connectivity.ps1 -Namespace contoso.servicebus.windows.net -Kafka
# Also probes Event Hubs Kafka port 9093
#>
param(
[string]$Namespace,
[switch]$Kafka
)
if ([string]::IsNullOrWhiteSpace($Namespace)) {
Write-Error "Namespace is required. Usage: test-messaging-connectivity.ps1 -Namespace <namespace> [-Kafka]"
exit 2
}
# Accept a bare namespace name or a full FQDN.
$fqdn = if ($Namespace -like "*.*") { $Namespace } else { "$Namespace.servicebus.windows.net" }
Write-Host "Testing messaging connectivity for: $fqdn"
Write-Host ""
# ── DNS resolution ────────────────────────────────────────────────────────────
$resolvedIp = $null
try {
$records = Resolve-DnsName -Name $fqdn -ErrorAction Stop
$resolvedIp = ($records | Where-Object { $_.IPAddress } | Select-Object -First 1).IPAddress
} catch {
try {
$resolvedIp = ([System.Net.Dns]::GetHostAddresses($fqdn) | Select-Object -First 1).IPAddressToString
} catch {
$resolvedIp = $null
}
}
$dnsOk = [bool]$resolvedIp
$dnsResult = if ($dnsOk) { "resolved ($resolvedIp)" } else { "NOT RESOLVED" }
# ── TCP port probe ────────────────────────────────────────────────────────────
function Test-TcpPort {
param([string]$TargetHost, [int]$Port)
try {
$client = [System.Net.Sockets.TcpClient]::new()
$async = $client.BeginConnect($TargetHost, $Port, $null, $null)
$ok = $async.AsyncWaitHandle.WaitOne(5000, $false)
if ($ok -and $client.Connected) {
$client.EndConnect($async)
$client.Close()
return $true
}
$client.Close()
return $false
} catch {
return $false
}
}
function Get-PortResult {
param([int]$Port)
if (Test-TcpPort -TargetHost $fqdn -Port $Port) { "reachable" } else { "BLOCKED" }
}
# ── HTTPS reachability ────────────────────────────────────────────────────────
# On success the namespace returns an Atom feed or HTTP 401 — either proves the
# endpoint is reachable. A connection failure means blocked. -SkipHttpErrorCheck
# is intentionally not used: it is unavailable in Windows PowerShell 5.1, so we
# instead treat an HTTP-error response (caught below) as proof of reachability.
function Get-HttpsResult {
try {
$resp = Invoke-WebRequest -Uri "https://$fqdn/" -Method Get -TimeoutSec 15 `
-UseBasicParsing -ErrorAction Stop
return "reachable (HTTP $($resp.StatusCode))"
} catch {
# An HTTP error response (e.g. 401) still proves reachability. The
# exception type differs between PowerShell editions (WebException in
# 5.1, HttpResponseException in 7+) but both expose Response.StatusCode.
$status = $_.Exception.Response.StatusCode
if ($null -ne $status) {
return "reachable (HTTP $([int]$status))"
}
# Fall back to a plain TCP probe of 443.
if (Test-TcpPort -TargetHost $fqdn -Port 443) {
return "reachable (TCP 443 open)"
}
return "BLOCKED"
}
}
$rows = [System.Collections.Generic.List[object]]::new()
$rows.Add([PSCustomObject]@{ Check = "DNS resolution"; Port = "-"; Result = $dnsResult })
if ($dnsOk) {
$rows.Add([PSCustomObject]@{ Check = "HTTPS reachability"; Port = "443"; Result = (Get-HttpsResult) })
$rows.Add([PSCustomObject]@{ Check = "AMQP over TLS"; Port = "5671"; Result = (Get-PortResult 5671) })
$rows.Add([PSCustomObject]@{ Check = "AMQP"; Port = "5672"; Result = (Get-PortResult 5672) })
$rows.Add([PSCustomObject]@{ Check = "HTTPS / WebSockets"; Port = "443"; Result = (Get-PortResult 443) })
if ($Kafka) {
$rows.Add([PSCustomObject]@{ Check = "Event Hubs Kafka"; Port = "9093"; Result = (Get-PortResult 9093) })
}
} else {
$rows.Add([PSCustomObject]@{ Check = "HTTPS reachability"; Port = "443"; Result = "skipped (DNS failed)" })
$rows.Add([PSCustomObject]@{ Check = "AMQP over TLS"; Port = "5671"; Result = "skipped (DNS failed)" })
$rows.Add([PSCustomObject]@{ Check = "AMQP"; Port = "5672"; Result = "skipped (DNS failed)" })
$rows.Add([PSCustomObject]@{ Check = "HTTPS / WebSockets"; Port = "443"; Result = "skipped (DNS failed)" })
if ($Kafka) {
$rows.Add([PSCustomObject]@{ Check = "Event Hubs Kafka"; Port = "9093"; Result = "skipped (DNS failed)" })
}
}
$rows | Format-Table -AutoSize
if (-not $dnsOk) {
Write-Host "Summary: could not resolve $fqdn. Check the namespace name and DNS/private-endpoint configuration before testing ports."
} else {
Write-Host "Summary: DNS resolved to $resolvedIp. 'reachable' ports accept TCP connections; any 'BLOCKED' port points to an IP firewall, NSG, corporate proxy, or private-endpoint restriction to investigate. Port 443 (WebSockets) can be used as a fallback when AMQP ports 5671/5672 are blocked."
}
test-messaging-connectivity.sh 6.3 KB
#!/usr/bin/env bash
# test-messaging-connectivity.sh
# Probes reachability of an Azure Service Bus / Event Hubs namespace and prints a
# normalized per-check report: DNS resolution, HTTPS reachability, and TCP
# connectivity to the well-known messaging ports (AMQP 5671/5672, HTTPS 443, and
# — with --kafka — Event Hubs Kafka 9093).
#
# A blocked port is a valid diagnostic result, not a failure: the script exits 0
# unless the arguments are invalid. Choosing which namespace to test and
# diagnosing a blocked port (IP firewall vs. corporate proxy vs. NSG) require
# judgment and stay in the skill prose.
#
# Usage:
# ./test-messaging-connectivity.sh <namespace> [--kafka]
#
# The namespace may be a full FQDN or a bare namespace name; when no dot is
# present, ".servicebus.windows.net" is appended automatically.
#
# Examples:
# ./test-messaging-connectivity.sh contoso # contoso.servicebus.windows.net
# ./test-messaging-connectivity.sh contoso.servicebus.windows.net # Service Bus / Event Hubs (AMQP + HTTPS)
# ./test-messaging-connectivity.sh contoso --kafka # also probe Event Hubs Kafka port 9093
set -uo pipefail
INCLUDE_KAFKA=0
NAMESPACE=""
while [ $# -gt 0 ]; do
case "$1" in
--kafka)
INCLUDE_KAFKA=1
;;
-h|--help)
grep '^#' "$0" | grep -v '^#!' | sed 's/^# \{0,1\}//'
exit 0
;;
--*)
echo "Unknown option: $1" >&2
echo "Usage: $0 <namespace> [--kafka]" >&2
exit 2
;;
*)
if [ -z "$NAMESPACE" ]; then
NAMESPACE="$1"
else
echo "Unexpected argument: $1" >&2
echo "Usage: $0 <namespace> [--kafka]" >&2
exit 2
fi
;;
esac
shift
done
if [ -z "$NAMESPACE" ]; then
echo "Usage: $0 <namespace> [--kafka]" >&2
exit 2
fi
# Accept a bare namespace name or a full FQDN.
FQDN="$NAMESPACE"
case "$FQDN" in
*.*) : ;; # already looks like an FQDN
*) FQDN="${FQDN}.servicebus.windows.net" ;;
esac
echo "Testing messaging connectivity for: $FQDN"
echo ""
# ── DNS resolution ────────────────────────────────────────────────────────────
resolve_ip() {
local host="$1" ip=""
if command -v getent >/dev/null 2>&1; then
ip=$(getent ahosts "$host" 2>/dev/null | awk '{print $1}' | head -n1)
fi
if [ -z "$ip" ] && command -v host >/dev/null 2>&1; then
ip=$(host "$host" 2>/dev/null | awk '/has address/ {print $NF; exit}')
fi
if [ -z "$ip" ] && command -v python3 >/dev/null 2>&1; then
ip=$(python3 -c "import socket,sys; print(socket.gethostbyname(sys.argv[1]))" "$host" 2>/dev/null)
fi
if [ -z "$ip" ] && command -v nslookup >/dev/null 2>&1; then
# Skip the leading "Server/Address" block; the answer's address follows
# the "Name:" line.
ip=$(nslookup "$host" 2>/dev/null | awk '/^Name:/ {seen=1; next} seen && /^Address/ {print $NF; exit}')
fi
printf '%s' "$ip"
}
RESOLVED_IP="$(resolve_ip "$FQDN")"
if [ -n "$RESOLVED_IP" ]; then
DNS_RESULT="resolved ($RESOLVED_IP)"
else
DNS_RESULT="NOT RESOLVED"
fi
# ── TCP port probe ────────────────────────────────────────────────────────────
probe_tcp() {
local host="$1" port="$2"
if command -v nc >/dev/null 2>&1; then
if nc -z -w 5 "$host" "$port" >/dev/null 2>&1; then
return 0
fi
return 1
fi
# Fallback: bash /dev/tcp with a background timeout.
( exec 3<>"/dev/tcp/$host/$port" ) >/dev/null 2>&1 &
local pid=$!
local waited=0
while kill -0 "$pid" 2>/dev/null; do
sleep 1
waited=$((waited + 1))
if [ "$waited" -ge 5 ]; then
kill "$pid" 2>/dev/null
wait "$pid" 2>/dev/null
return 1
fi
done
wait "$pid"
return $?
}
port_result() {
if probe_tcp "$FQDN" "$1"; then
echo "reachable"
else
echo "BLOCKED"
fi
}
# ── HTTPS reachability ────────────────────────────────────────────────────────
# On success the namespace returns an Atom feed or HTTP 401 — either proves the
# endpoint is reachable. A connection failure (curl exit != 0) means blocked.
https_result() {
if ! command -v curl >/dev/null 2>&1; then
# No curl: fall back to a plain TCP probe of 443.
if probe_tcp "$FQDN" 443; then
echo "reachable (TCP 443 open; curl unavailable for HTTP check)"
else
echo "BLOCKED"
fi
return
fi
local code
code=$(curl -s -o /dev/null -m 15 -w '%{http_code}' "https://$FQDN/" 2>/dev/null)
local rc=$?
if [ "$rc" -eq 0 ] && [ -n "$code" ] && [ "$code" != "000" ]; then
echo "reachable (HTTP $code)"
else
echo "BLOCKED (curl exit $rc)"
fi
}
DNS_OK=0; [ -n "$RESOLVED_IP" ] && DNS_OK=1
if [ "$DNS_OK" -eq 1 ]; then
HTTPS_RESULT="$(https_result)"
P443="$(port_result 443)"
P5671="$(port_result 5671)"
P5672="$(port_result 5672)"
if [ "$INCLUDE_KAFKA" -eq 1 ]; then
P9093="$(port_result 9093)"
fi
else
HTTPS_RESULT="skipped (DNS failed)"
P443="skipped (DNS failed)"
P5671="skipped (DNS failed)"
P5672="skipped (DNS failed)"
P9093="skipped (DNS failed)"
fi
# ── Report ────────────────────────────────────────────────────────────────────
printf '%-28s %-10s %s\n' "Check" "Port" "Result"
printf '%-28s %-10s %s\n' "-----" "----" "------"
printf '%-28s %-10s %s\n' "DNS resolution" "-" "$DNS_RESULT"
printf '%-28s %-10s %s\n' "HTTPS reachability" "443" "$HTTPS_RESULT"
printf '%-28s %-10s %s\n' "AMQP over TLS" "5671" "$P5671"
printf '%-28s %-10s %s\n' "AMQP" "5672" "$P5672"
printf '%-28s %-10s %s\n' "HTTPS / WebSockets" "443" "$P443"
if [ "$INCLUDE_KAFKA" -eq 1 ]; then
printf '%-28s %-10s %s\n' "Event Hubs Kafka" "9093" "$P9093"
fi
echo ""
if [ "$DNS_OK" -eq 0 ]; then
echo "Summary: could not resolve $FQDN. Check the namespace name and DNS/private-endpoint configuration before testing ports."
else
echo "Summary: DNS resolved to $RESOLVED_IP. 'reachable' ports accept TCP connections; any 'BLOCKED' port points to an IP firewall, NSG, corporate proxy, or private-endpoint restriction to investigate. Port 443 (WebSockets) can be used as a fallback when AMQP ports 5671/5672 are blocked."
fi
troubleshooting/aks/
aks-troubleshooting.md 6.2 KB
# AKS Troubleshooting Guide
Primary AKS troubleshooting guide for incidents routed from [../../SKILL.md](../../SKILL.md).
## When to Use This Guide
- lifecycle, access, node, `kube-system`, workload, ingress, DNS, or scaling issues
- `kubectl` cannot connect, nodes are `NotReady`, or pods are unhealthy
## Scenario Playbooks
| Scenario | Reference |
| ------------------------------------------------------------- | ------------------------------------------------ |
| broad cluster investigation | [general-diagnostics.md](general-diagnostics.md) |
| workload, crash, image pull, readiness, or pending pod issues | [pod-failures.md](pod-failures.md) |
| node health, scaling, pressure, upgrade, or zone issues | [node-issues.md](node-issues.md) |
| service, ingress, DNS, or network policy issues | [networking.md](networking.md) |
## Tool Selection For Diagnostics
When gathering AKS diagnostic evidence, prefer `mcp_azure_mcp_aks`, then the smallest discovered AKS-MCP tool that fits the read, then supporting Azure tools such as `mcp_azure_mcp_applens`, `mcp_azure_mcp_monitor`, or `mcp_azure_mcp_resourcehealth`. Use raw `az aks` and `kubectl` only when the AKS-MCP surface cannot perform the needed check.
When standard diagnostics do not reveal root cause, use **Inspektor Gadget** for real-time, low-level node and pod observability (DNS traces, TCP traces, process snapshots, file access traces). See [references/inspektor-gadget.md](references/inspektor-gadget.md) for the gadget catalog, the `run-ig` script, and symptom-to-gadget mapping.
See [references/aks-mcp.md](references/aks-mcp.md), [references/structured-input-modes.md](references/structured-input-modes.md), [references/command-flows.md](references/command-flows.md)
## Required Inputs
- subscription or active Azure context
- resource group and cluster name
- symptom summary
- first observed time or recent change window
- impacted namespace, workload, service, or ingress when known
If cluster identity is missing, stop and ask for it.
## Scope Buckets
- Lifecycle: create, update, start, stop, upgrade, or provisioning failures
- API access: kubeconfig, auth, private endpoint, DNS, or reachability problems
- Nodes: missing nodes, `NotReady`, pressure, CNI, kubelet, certificate, or VMSS drift
- `kube-system`: CoreDNS, metrics-server, konnectivity, ingress, CNI, CSI, or add-on failures
- Workloads: `Pending`, `CrashLoopBackOff`, `OOMKilled`, PVC, quota, secret, readiness, or dependency issues
- Connectivity and DNS: pod -> service -> endpoints -> ingress/load balancer -> DNS -> network controls
- Scaling: node pool sizing, pending pods, autoscaler config, metrics, quota, or subnet constraints
## Evidence Order
1. Azure-side state first: cluster state, resource health, recent operations, node pool state, detector or monitoring output.
2. Kubernetes-side state second: cluster reachability, nodes, `kube-system`, events, affected namespace, pod detail, logs.
3. Use detector, warning-event, or metrics modes when the incoming data already matches them.
4. Deep diagnostics; when steps 1–3 do not reveal root cause, use [inspektor-gadget.md](references/inspektor-gadget.md) for real-time tracing and snapshots on the affected node.
## Workflow
1. Get cluster context.
2. Classify the problem by scope bucket.
3. Prefer Azure-side evidence before Kubernetes-side evidence.
4. Use the matching AKS-MCP path first, then the documented CLI fallback if MCP cannot perform that read.
5. Return evidence, failure domain, confidence, next checks, remediation, and escalation.
## Error Patterns
- No cluster context: ask for subscription, resource group, and cluster name.
- MCP unavailable: fall back to safe `az aks` and `kubectl` reads.
- `kubectl` blocked: separate auth problems from network reachability.
- Logs or metrics missing: use events, node state, and resource descriptions.
- Detector noise: ignore `emergingIssues`, prefer critical findings, rank the most actionable signal first.
## Safe Fallback Checks
When AKS-MCP cannot perform the baseline read, run the **[`aks-baseline`](../../scripts/aks-baseline.sh)** script. It executes the read-only cluster + Kubernetes baseline sweep (provisioning state, node pools, activity log, node readiness, unhealthy pods, kube-system health, warning events) and returns a single labeled digest:
```bash
# bash
./scripts/aks-baseline.sh -g <resource-group> -n <cluster-name> [--namespace <namespace>]
```
```powershell
# PowerShell
.\scripts\aks-baseline.ps1 -ResourceGroup <resource-group> -Cluster <cluster-name> [-Namespace <namespace>]
```
Then deep-dive on a specific pod as the digest indicates:
```bash
az aks show -g <resource-group> -n <cluster-name>
az aks nodepool list -g <resource-group> --cluster-name <cluster-name>
kubectl cluster-info
kubectl get nodes -o wide
kubectl get pods -n kube-system
kubectl get events -A --sort-by=.lastTimestamp
kubectl describe pod <pod-name> -n <namespace>
kubectl logs <pod-name> -n <namespace> --previous
```
For unhealthy pods, gather the full read-only evidence bundle (describe, current + previous logs, resources vs usage) with the pod-evidence script instead of running the commands one by one — [`../../scripts/pod-evidence.sh`](../../scripts/pod-evidence.sh) / [`../../scripts/pod-evidence.ps1`](../../scripts/pod-evidence.ps1):
```bash
../../scripts/pod-evidence.sh <pod-name> -n <namespace>
../../scripts/pod-evidence.sh --all-failing
```
```powershell
../../scripts/pod-evidence.ps1 <pod-name> -Namespace <namespace>
../../scripts/pod-evidence.ps1 -AllFailing
```
See [pod-failures.md](pod-failures.md) for how to interpret the digest.
Keep these read-only unless the user explicitly asks for remediation.
## Guardrails
- default to read-only diagnostics
- do not restart, delete, cordon, drain, scale, upgrade, or reconfigure resources unless the user explicitly asks for remediation
- do not conclude root cause without quoting the evidence that supports it
## Output Checklist
Return scope and impact, evidence, failure domain, root cause, confidence, next checks, remediation, and escalation.
general-diagnostics.md 2.0 KB
# General AKS Investigation & Diagnostics
## "What happened in my cluster?"
When a user asks a broad question like "what happened in my AKS cluster?" or "check my AKS status", follow this systematic flow:
1. Cluster health
2. Recent events
3. Node status
4. Unhealthy pods
5. All pods overview
6. System pods health
7. Activity log
Run the **[`aks-baseline`](../../scripts/aks-baseline.sh)** script instead of issuing these commands one by one. It performs the entire read-only sweep above and prints a single labeled digest (provisioning state, node pool summary, recent activity log, node readiness, unhealthy pods, kube-system health, and recent warning events), so you get one summarized result instead of seven raw dumps.
```bash
# bash
./scripts/aks-baseline.sh -g <rg> -n <cluster> [--namespace <ns>]
```
```powershell
# PowerShell
.\scripts\aks-baseline.ps1 -ResourceGroup <rg> -Cluster <cluster> [-Namespace <ns>]
```
After reviewing the digest, deep-dive into a specific pod with `kubectl describe` / `kubectl logs`.
---
## AKS CLI Tools
```bash
# Get cluster credentials (required before kubectl commands)
az aks get-credentials -g <rg> -n <cluster>
# View node pools
az aks nodepool list -g <rg> --cluster-name <cluster> -o table
```
### AppLens (MCP) for AKS
For AI-powered diagnostics:
```text
mcp_azure_mcp_applens
intent: "diagnose AKS cluster issues"
command: "diagnose"
parameters:
resourceId: "/subscriptions/<sub>/resourceGroups/<rg>/providers/Microsoft.ContainerService/managedClusters/<cluster>"
```
> 💡 **Tip:** AppLens automatically detects common issues and provides remediation recommendations using the cluster resource ID.
---
## Best Practices
1. **Start with kubectl get/describe** - Always check basic status first
2. **Check events** - `kubectl get events -A` reveals recent issues
3. **Use systematic isolation** - Pod -> Node -> Cluster -> Network
4. **Document changes** - Note what you tried and what worked
5. **Escalate when needed** - For control plane issues, contact Azure support
load-balancer-and-ingress.md 3.6 KB
# Load Balancer And Ingress Troubleshooting
Use this guide when AKS networking symptoms point at Azure load balancer provisioning, ingress controller behavior, or backend routing.
## Load Balancer Stuck In Pending
**Diagnostics:**
```bash
kubectl describe svc <svc> -n <ns>
# Events section reveals the actual Azure error
kubectl logs -n kube-system -l component=cloud-controller-manager --tail=100
```
**Error decision table:**
| Error in Events / CCM Logs | Cause | Fix |
| ------------------------------------------------------ | -------------------------------------- | ---------------------------------------------------------------------------- |
| `InsufficientFreeAddresses` | Subnet has no free IPs | Expand subnet CIDR; use Azure CNI Overlay; use NAT gateway instead |
| `ensure(default/svc): failed... PublicIPAddress quota` | Public IP quota exhausted | Request quota increase for Public IP Addresses in the region |
| `cannot find NSG` | NSG name changed or detached | Re-associate NSG to the AKS subnet; check `az aks show` for NSG name |
| `reconciling NSG rules: failed` | NSG is locked or has conflicting rules | Remove resource lock; check for deny-all rules above AKS-managed rules |
| `subnet not found` | Wrong subnet name in annotation | Verify subnet name: `az network vnet subnet list -g <rg> --vnet-name <vnet>` |
| No events, stuck Pending | CCM can't authenticate to Azure | Check cluster managed identity access on the VNet resource group |
---
## Ingress Not Routing Traffic
**Diagnostics:**
```bash
# Confirm controller is running
kubectl get pods -n <ingress-ns> -l 'app.kubernetes.io/name in (ingress-nginx,nginx-ingress)'
kubectl logs -n <ingress-ns> -l app.kubernetes.io/name=ingress-nginx --tail=100
# Check the ingress resource state
kubectl describe ingress <name> -n <ns>
kubectl get ingress <name> -n <ns>
# Check backend
kubectl get endpoints <backend-svc> -n <ns>
```
**Ingress failure patterns:**
| Symptom | Cause | Fix |
| -------------------------------- | ---------------------------------------------- | ------------------------------------------------------------ |
| ADDRESS empty | LB not provisioned or wrong `ingressClassName` | Check controller service; set correct `ingressClassName` |
| 404 for all paths | No matching host rule | Check `host` field; `pathType: Prefix` vs `Exact` |
| 404 for some paths | Trailing slash mismatch | `Prefix /api` matches `/api/foo` not `/api` - add both |
| 502 Bad Gateway | Backend pods unhealthy or wrong port | Verify Endpoints has IPs; confirm `targetPort` and readiness |
| 503 Service Unavailable | All backend pods down | Check pod restarts and readiness probe |
| TLS handshake fail | cert-manager not issuing | Check certificate status and ACME challenge |
| Works for host-a, 404 for host-b | DNS not pointing to ingress IP | Verify `nslookup <host>` resolves to the ingress address |
network-policy.md 0.8 KB
# Network Policy Troubleshooting
Use this guide when pod-to-pod or pod-to-service traffic is selectively blocked and the symptom points at ingress or egress filtering.
```bash
# List all policies in the namespace - check both ingress and egress
kubectl get networkpolicy -n <ns> -o yaml
# Check for a default-deny policy (blocks everything unless explicitly allowed)
kubectl get networkpolicy -n <ns> -o jsonpath='{range .items[?(@.spec.podSelector=={})]}{.metadata.name}{"\n"}{end}'
```
**AKS network policy engine check:** Azure NPM (Azure CNI): `kubectl get pods -n kube-system -l k8s-app=azure-npm`. Calico: `kubectl get pods -n calico-system`.
Policy audit: source labels, destination labels, destination ingress rules, and source egress rules must all line up. With default-deny, explicitly allow UDP/TCP 53 to kube-dns.
networking.md 5.5 KB
# Networking Troubleshooting
For CNI-specific issues, check CNI pod health and review [AKS networking concepts](https://learn.microsoft.com/azure/aks/concepts-network).
## Service Unreachable / Connection Refused
**Diagnostics - always start here:**
```bash
# 1. Verify service exists and has endpoints (read-only)
kubectl get svc <service-name> -n <ns>
kubectl get endpoints <service-name> -n <ns>
# 2. Optional connectivity test from inside the namespace
# This creates a temporary pod. Prefer read-only checks first.
# Only use it after the user explicitly approves a mutating test.
kubectl run netdebug --image=curlimages/curl -it --rm -n <ns> -- \
curl -sv http://<service>.<ns>.svc.cluster.local:<port>/healthz
```
**Decision tree:**
| Observation | Cause | Fix |
| --------------------------------------- | ---------------------------------- | ----------------------------------------------- |
| Endpoints shows `<none>` | Label selector mismatch | Align selector with pod labels; check for typos |
| Endpoints has IPs but unreachable | Port mismatch or app not listening | Confirm `targetPort` = actual container port |
| Works from some pods, fails from others | Network policy blocking | See Network Policy section |
| Works inside cluster, fails externally | Load balancer issue | See Load Balancer section |
| `ECONNREFUSED` immediately | App not listening on that port | Check listening ports in the pod |
Pods that are running but not Ready are removed from Endpoints. Check `kubectl get pod <pod> -n <ns>`.
**Deep diagnostics with Inspektor Gadget** (when the above checks are inconclusive):
Use [`scripts/run-ig.sh`](references/inspektor-gadget.md) (or `run-ig.ps1`) with `--pod <pod-name> --ns <ns>` and these gadgets:
- `snapshot_socket` — check what ports the pod is listening on
- `trace_tcp` — trace connect/accept/close events
- `trace_tcpretrans` — packet retransmissions
See [references/inspektor-gadget.md](references/inspektor-gadget.md).
---
## DNS Resolution Failures
**Diagnostics:**
The live DNS test creates a temporary pod. Prefer `get`, `describe`, `logs`, or `exec` into an existing pod first. Only use it after the user explicitly approves creating the test pod.
```bash
# Confirm CoreDNS is running and healthy (read-only)
kubectl get pods -n kube-system -l k8s-app=kube-dns -o wide
kubectl top pod -n kube-system -l k8s-app=kube-dns
# Optional live DNS test from the same namespace as the failing pod
kubectl run dnstest --image=busybox:1.28 -it --rm -n <ns> -- \
nslookup <service-name>.<ns>.svc.cluster.local
# CoreDNS logs - errors show here first
kubectl logs -n kube-system -l k8s-app=kube-dns --tail=100
```
**DNS failure patterns:**
| Symptom | Cause | Fix |
| ------------------------------------- | -------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------- |
| `NXDOMAIN` for `svc.cluster.local` | CoreDNS down or pod network broken | After confirming the diagnostics above, coordinate with the cluster operator to restart or redeploy CoreDNS and verify CNI |
| Internal resolves, external NXDOMAIN | Custom DNS not forwarding to `168.63.129.16` | Fix upstream forwarder |
| Intermittent SERVFAIL under load | CoreDNS CPU throttled | Remove CPU limits or add replicas |
| Private cluster - external names fail | Custom DNS missing privatelink forwarder | Add conditional forwarder to Azure DNS |
| `i/o timeout` not `NXDOMAIN` | Port 53 blocked by NetworkPolicy or NSG | Allow UDP/TCP 53 from pods to kube-dns ClusterIP |
> ⚠️ **Warning:** The fixes in this table can change cluster state. Use them only after performing the read-only diagnostics above, and only with explicit confirmation from the cluster owner or operator.
```bash
kubectl get svc kube-dns -n kube-system -o jsonpath='{.spec.clusterIP}'
```
Custom VNet DNS must forward `.cluster.local` to the CoreDNS ClusterIP and other lookups to `168.63.129.16`.
**Deep diagnostics with Inspektor Gadget** (when the above checks are inconclusive):
Use [`scripts/run-ig.sh`](references/inspektor-gadget.md) (or `run-ig.ps1`) with `--pod <pod-name> --ns <ns>` and `trace_dns`. Key signals: `rcode=3` (NXDOMAIN), `rcode=2` (SERVFAIL), high `latency` values, queries going to unexpected destinations.
See [references/inspektor-gadget.md](references/inspektor-gadget.md).
---
## Detailed Networking Guides
- [Load Balancer And Ingress Troubleshooting](load-balancer-and-ingress.md) for pending services, ingress controller state, backend routing, and TLS failures.
- [Network Policy Troubleshooting](network-policy.md) for default-deny checks, Azure NPM or Calico validation, and ingress or egress rule audits.
node-issues.md 4.7 KB
# Node & Cluster Troubleshooting
## Node NotReady
**Diagnostics:**
```bash
kubectl get nodes -o wide
kubectl describe node <node-name>
# Look for: Conditions, Taints, Events, resource usage, kubelet status
```
**Condition decision tree:**
| Condition | Value | Meaning | Fix Path |
| -------------------- | ------- | --------------------------------- | ------------------------------------------------------------- |
| `Ready` | `False` | kubelet stopped reporting | SSH to node; if unrecoverable, consider cordon/drain/delete\* |
| `MemoryPressure` | `True` | Node running out of memory | Evict pods; scale out pool; reduce pod density |
| `DiskPressure` | `True` | OS disk or ephemeral storage full | Check logs and images; clean up or increase disk |
| `PIDPressure` | `True` | Too many processes | App spawning excessive threads/processes; use IG `snapshot_process` |
| `NetworkUnavailable` | `True` | CNI plugin issue | Check CNI pods in kube-system; node network config |
\*Only after explicit user request for remediation and confirmation of workload impact.
**AKS-specific - SSH to a node:**
> ⚠️ **Warning:** `kubectl debug node/...` creates a privileged debug pod on the node and is not a read-only diagnostic step. Default to read-only evidence gathering first. Only suggest or run this after the user explicitly asks for remediation or approves a privileged diagnostic action and understands the change-control impact.
```bash
# Create a privileged debug pod on the node
kubectl debug node/<node-name> -it --image=mcr.microsoft.com/cbl-mariner/base/core:2.0
# Check kubelet status inside the node
chroot /host systemctl status kubelet
chroot /host journalctl -u kubelet -n 50
```
**Optional remediation if kubelet can't recover (after confirmation):** cordon -> drain -> delete. AKS auto-replaces via node pool VMSS.
> ⚠️ **Warning:** These commands are disruptive. By default, stay in read-only diagnostic mode. Only suggest or run them if the user has explicitly requested remediation and confirmed they understand the workload and PodDisruptionBudget impact.
```bash
kubectl cordon <node-name>
kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data
kubectl delete node <node-name>
```
---
## Node Pool Not Scaling
### Cluster Autoscaler Not Triggering
**Diagnostics:**
```bash
# Autoscaler logs
kubectl logs -n kube-system -l app=cluster-autoscaler --tail=100
# Autoscaler status
kubectl get configmap cluster-autoscaler-status -n kube-system -o yaml
# Verify autoscaler is enabled on the node pool
az aks nodepool show -g <rg> --cluster-name <cluster> -n <nodepool> \
--query "{autoscaleEnabled:enableAutoScaling, min:minCount, max:maxCount}"
```
**Autoscaler won't scale up - common reasons:**
- Node pool already at `maxCount`
- VM quota exhausted: `az vm list-usage -l <region> -o table | grep -i "DSv3\|quota"`
- Pod `nodeAffinity` is unsatisfiable on any new node template
- 10-minute cooldown period still active after last scale event
**Autoscaler won't scale down - common reasons:**
- Pods with `emptyDir` local storage (configure `--skip-nodes-with-local-storage=false` if safe)
- Standalone pods with no controller (not in a ReplicaSet)
- `cluster-autoscaler.kubernetes.io/safe-to-evict: "false"` annotation on a pod
### Manual Scaling
```bash
az aks nodepool scale -g <rg> --cluster-name <cluster> -n <nodepool> --node-count <n>
```
---
## Resource Pressure & Capacity Planning
**Check actual vs allocatable:**
```bash
kubectl describe node <node> | grep -A6 "Allocated resources:"
```
See [AKS resource reservations](https://learn.microsoft.com/azure/aks/concepts-clusters-workloads#resource-reservations) for allocatable math.
**Ephemeral storage pressure:**
```bash
# Check what's consuming ephemeral storage on a node
kubectl debug node/<node> -it --image=mcr.microsoft.com/cbl-mariner/base/core:2.0
```
Common culprit: high-volume container logs accumulating in `/var/log/containers`.
**Deep diagnostics with Inspektor Gadget** (PID pressure or unknown process load):
Use `scripts/run-ig.sh --gadget snapshot_process --node <node-name>` (or `run-ig.ps1`) to list all processes on the node. For node-wide scope, use `--node` (no pod filters). See [references/inspektor-gadget.md](references/inspektor-gadget.md).
---
## Detailed Node And Cluster Guides
- [Upgrade Operations](upgrade-operations.md) for node images, Kubernetes version upgrades, surge settings, and PDB-related drain blockers.
- [Spot And Zone Issues](spot-and-zone-issues.md) for spot evictions, tolerations, zone skew, and zonal storage or service behavior.
pod-failures.md 7.5 KB
# Pod Failures & Application Issues
## Evidence Bundle Script
For **any** pod symptom below, run the **pod-evidence** script to collect the same
read-only evidence bundle. Per pod it digests **STATUS**, **STATE** (exit code, reason,
last state), **EVENTS**, current/previous **LOGS**, and **RESOURCES** (requests vs
`top`). It only gathers; interpret with the tables.
Bash [`../../scripts/pod-evidence.sh`](../../scripts/pod-evidence.sh) · PowerShell [`../../scripts/pod-evidence.ps1`](../../scripts/pod-evidence.ps1)
```bash
../../scripts/pod-evidence.sh <pod-name> -n <namespace> # one pod
../../scripts/pod-evidence.sh --all-failing # all unhealthy pods
```
PowerShell: `../../scripts/pod-evidence.ps1 <pod-name> -Namespace <namespace>` (`-AllFailing` scans all).
---
## CrashLoopBackOff
Pod starts, crashes, restarts with exponential backoff (10s, 20s, 40s... up to 5m).
**Diagnostics:** [pod-evidence](#evidence-bundle-script) → read **STATE** (exit code, reason, last state) and **PREV LOGS** (last crashed container).
**Decision tree:**
| Exit Code | Meaning | Fix Path |
| --------- | ----------------------------------------------------- | ------------------------------------------------------------- |
| `0` | App exited successfully (unexpected for long-running) | Check if entrypoint/command is correct; app may be a one-shot |
| `1` | Application error | Read logs - unhandled exception, missing config, bad startup |
| `137` | OOMKilled (SIGKILL) | Increase `resources.limits.memory`; check for memory leaks |
| `139` | Segfault (SIGSEGV) | Binary compatibility issue or native code bug |
| `143` | SIGTERM - graceful shutdown | Pod was terminated; check if liveness probe killed it |
**OOMKilled specifically:** the **STATE** section shows `terminated=OOMKilled` and **RESOURCES** shows the memory limit vs live usage. Fix: increase `resources.limits.memory` or optimize application memory usage.
**OOM kill tracing with Inspektor Gadget:** Run `trace_oomkill` for the pod to see which process was killed and memory at kill time: `scripts/run-ig.sh --gadget trace_oomkill --pod <pod-name> --ns <namespace>` (or `run-ig.ps1`).
**Deep diagnostics with Inspektor Gadget** (when logs and describe are inconclusive):
Use [`scripts/run-ig.sh`](references/inspektor-gadget.md) (or `run-ig.ps1`) with `--pod <pod-name> --ns <namespace>` and these gadgets:
- `trace_exec` — see what the container executes at startup
- `trace_open` — find missing configs/secrets (retval -2 = ENOENT, -13 = EACCES)
- `snapshot_process` — list running processes in the pod
---
## ImagePullBackOff
Pod can't pull the container image.
**Diagnostics:** [pod-evidence](#evidence-bundle-script) → read **EVENTS** for the exact pull error.
| Error Message | Cause | Fix |
| --------------------------------------- | ---------------------------- | -------------------------------------------------------------- |
| `ErrImagePull` / `ImagePullBackOff` | Image name or tag is wrong | Verify image name and tag exist in the registry |
| `unauthorized: authentication required` | Missing or wrong pull secret | Create/update `imagePullSecrets` on the pod or service account |
| `manifest unknown` | Tag doesn't exist | Check available tags in the registry |
| `context deadline exceeded` | Registry unreachable | Check network/firewall; for ACR, verify AKS -> ACR integration |
**ACR integration check:**
```bash
# Verify AKS is attached to ACR
az aks check-acr -g <rg> -n <cluster> --acr <acr-name>.azurecr.io
```
---
## Pending Pods
Pod stays in `Pending` - scheduler can't place it.
**Diagnostics:** [pod-evidence](#evidence-bundle-script) → read **EVENTS** for why scheduling failed.
| Event Message | Cause | Fix |
| ---------------------------------------------------------------------- | ----------------------------------- | --------------------------------------------------------------- |
| `Insufficient cpu` / `Insufficient memory` | No node has enough resources | Scale node pool; reduce resource requests; check for overcommit |
| `node(s) had taint ... that the pod didn't tolerate` | Taint/toleration mismatch | Add matching toleration or use a different node pool |
| `node(s) didn't match Pod's node affinity/selector` | Affinity rule unsatisfiable | Check `nodeSelector` or `nodeAffinity` rules |
| `persistentvolumeclaim ... not found` / `unbound` | PVC not ready | Check PVC status; verify storage class exists |
| `0/N nodes are available: N node(s) had volume node affinity conflict` | Zonal disk vs pod in different zone | Use ZRS storage class or ensure same zone |
---
## Readiness & Liveness Probe Failures
**Readiness probe failure** -> pod removed from Service endpoints (no traffic). **Liveness probe failure** -> pod killed and restarted.
**Diagnostics:** [pod-evidence](#evidence-bundle-script) → **EVENTS** shows `Readiness/Liveness probe failed`; **STATUS** shows the READY column (must be n/n).
| Symptom | Cause | Fix |
| ------------------------------------ | ----------------------- | ---------------------------------------------------------- |
| READY shows `0/1` but pod is Running | Readiness probe failing | Check probe path, port, and app health endpoint |
| Pod restarts repeatedly | Liveness probe failing | Increase `initialDelaySeconds`; check if app starts slowly |
| Probe timeout errors | App responds too slowly | Increase `timeoutSeconds`; check app performance |
> 💡 **Tip:** Set `initialDelaySeconds` on liveness probes to be longer than your app's startup time. A common mistake is killing pods before they finish initializing.
---
## Resource Constraints (CPU/Memory)
**Check actual usage vs limits:** [pod-evidence](#evidence-bundle-script) → **RESOURCES** compares requests/limits against live `top` usage. To rank a namespace by memory: `kubectl top pod -n <namespace> --sort-by=memory`.
| Symptom | Cause | Fix |
| -------------------------------- | --------------------------------------- | --------------------------------------------------- |
| OOMKilled (exit code 137) | Container exceeded memory limit | Increase `limits.memory` or fix memory leak |
| CPU throttling (slow responses) | Container hitting CPU limit | Increase `limits.cpu` or remove CPU limits |
| Pending - insufficient resources | Requests exceed available node capacity | Lower requests, scale nodes, or use larger VM sizes |
> ⚠️ **Warning:** Setting CPU limits can cause unnecessary throttling even when the node has spare capacity. Many teams set CPU requests but not limits. Memory limits should always be set.
spot-and-zone-issues.md 2.5 KB
# Spot And Zone Issues
Use this guide when workload placement, evictions, or zonal behavior is causing node-pool instability.
## Spot Node Pool Evictions
AKS spot nodes use Azure Spot VMs - they can be evicted with 30 seconds notice when Azure needs capacity.
**Diagnose spot eviction:**
```bash
# Spot nodes carry this taint automatically
kubectl describe node <node> | grep "Taint"
# kubernetes.azure.com/scalesetpriority=spot:NoSchedule
# Check eviction events
kubectl get events -A --field-selector reason=SpotEviction
kubectl get events -A | grep -i "evict\|spot\|preempt"
```
**Spot workload pattern:** pods must tolerate the spot taint. Prefer PDBs and avoid stateful PVC workloads on spot.
```yaml
tolerations:
- key: "kubernetes.azure.com/scalesetpriority"
operator: Equal
value: spot
effect: NoSchedule
```
Add this preferred node affinity when you want the workload to bias toward spot nodes:
```yaml
affinity:
nodeAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 1
preference:
matchExpressions:
- key: kubernetes.azure.com/scalesetpriority
operator: In
values: ["spot"]
```
---
## Multi-AZ Node Pool & Zone-Related Failures
**Check zone distribution:**
```bash
kubectl get nodes -L topology.kubernetes.io/zone
```
**Zone-related failure patterns:**
| Symptom | Cause | Fix |
| ------------------------------------------------ | ---------------------------------------------------- | ------------------------------------------------------------ |
| Pods stack on one zone after node failures | Scheduling imbalance after zone failure | `kubectl rollout restart deployment/<n>` to rebalance |
| PVC pending with `volume node affinity conflict` | Azure Disk is zonal; pod scheduled in different zone | Use ZRS storage class or ensure PVC and pod are in same zone |
| Service endpoints unreachable from one zone | Topology-aware routing misconfigured | Check `service.spec.trafficDistribution` or TopologyKeys |
| Upgrade causing zone imbalance | Surge nodes in one zone | Configure `maxSurge` in node pool upgrade settings |
Use `Premium_ZRS` or `StandardSSD_ZRS` in custom StorageClasses to reduce zonal PVC conflicts. See [AKS storage best practices](https://learn.microsoft.com/azure/aks/operator-best-practices-storage).
upgrade-operations.md 2.2 KB
# Upgrade Operations
Use this guide when node image rotation, Kubernetes version changes, or node-pool upgrade settings appear to be the failure domain.
## Node Image / OS Upgrade Issues
> ⚠️ **Warning:** `az aks nodepool upgrade` and `az aks nodepool update --max-surge ...` change cluster state. During diagnostics, do not recommend or run upgrade actions by default. Only surface these commands after the user explicitly approves remediation or confirms the change window / change-control context.
```bash
# Check current node image versions
az aks nodepool show -g <rg> --cluster-name <cluster> -n <nodepool> \
--query "{nodeImageVersion:nodeImageVersion, osType:osType}"
# Check available upgrades
az aks nodepool get-upgrades -g <rg> --cluster-name <cluster> --nodepool-name <nodepool>
# Upgrade node image (non-disruptive with surge)
az aks nodepool upgrade -g <rg> --cluster-name <cluster> -n <nodepool> --node-image-only
```
---
## Kubernetes Version Upgrade Failures
**Pre-upgrade check:**
```bash
# Check for deprecated API usage before upgrading
kubectl get --raw /metrics | grep apiserver_requested_deprecated_apis
# Verify available upgrade paths (can only skip one minor version)
az aks get-upgrades -g <rg> -n <cluster> -o table
```
**Upgrade stuck or failed:**
```bash
# Check control plane provisioning state
az aks show -g <rg> -n <cluster> --query "provisioningState"
# If stuck: check AKS diagnostics blade in portal
# Azure Portal -> AKS cluster -> Diagnose and solve problems -> Upgrade
```
Common causes: PDB blocking drain (`kubectl get pdb -A`), deprecated APIs in use, custom admission webhooks failing (`kubectl get validatingwebhookconfiguration`).
---
## Zero-Downtime Node Pool Upgrades
`maxSurge` controls how many extra nodes are provisioned during upgrade.
```bash
# Check current maxSurge
az aks nodepool show -g <rg> --cluster-name <cluster> -n <nodepool> \
--query "upgradeSettings.maxSurge"
az aks nodepool update -g <rg> --cluster-name <cluster> -n <nodepool> \
--max-surge 33%
```
**Upgrade stuck / nodes not draining:**
```bash
kubectl get pdb -A
kubectl describe pdb <pdb-name> -n <ns>
```
If `DisruptionsAllowed: 0`, scale up the workload or temporarily relax `minAvailable`.
troubleshooting/aks/references/
aks-mcp.md 1.5 KB
# AKS MCP Reference
Use this reference when AKS-aware MCP tools are available in the client.
## Preference Order
1. `mcp_azure_mcp_aks`
2. The AKS-MCP tools that surface after discovery in the client
3. Supporting Azure tools such as `mcp_azure_mcp_applens`, `mcp_azure_mcp_monitor`, and `mcp_azure_mcp_resourcehealth`
4. Raw `az aks` and `kubectl` only when required functionality is missing from MCP
## Happy Path
After selecting `mcp_azure_mcp_aks`, let the client enumerate the exact AKS-MCP tools it exposes and choose the smallest tool that fits the task.
Favor the obvious read paths first:
- cluster and Azure-side inspection
- detector or diagnostic workflows
- monitoring, metrics, or control-plane-log checks
- kubectl-style read operations
## Authentication And Access
AKS-MCP is Azure CLI-backed. Expect service principal, workload identity, managed identity, or existing `az login` auth, usually keyed by `AZURE_CLIENT_ID`. If `AZURE_SUBSCRIPTION_ID` is set, expect the server to select that subscription after login.
Default to `readonly`. Only suggest `readwrite` or `admin` when the current diagnostic step truly requires it.
## Detector Notes
For detector-style workflows, use the cluster resource ID, keep the time window within the last 30 days, cap each run to 24 hours, and stay within the supported AKS detector categories.
## Fallback Rule
If the client does not expose the AKS-MCP surface needed for a check, then fall back to:
- `az aks` for Azure-side AKS operations
- raw `kubectl` for Kubernetes-side inspection
command-flows.md 4.0 KB
# AKS Command Flows
## Cluster Baseline Flow
```text
Resolve subscription -> resolve resource group -> resolve cluster -> inspect cluster state -> inspect node pools -> inspect resource health -> inspect recent operations
```
CLI fallback when AKS-MCP cannot perform the cluster baseline read — run the **[`aks-baseline`](../../../scripts/aks-baseline.sh)** script, which gathers cluster state, node pools, and recent operations as one read-only digest:
```bash
# bash
./scripts/aks-baseline.sh -g <resource-group> -n <cluster-name>
```
```powershell
# PowerShell
.\scripts\aks-baseline.ps1 -ResourceGroup <resource-group> -Cluster <cluster-name>
```
## Kubernetes Baseline Flow
```text
Check API reachability -> inspect nodes -> inspect kube-system -> inspect events -> inspect affected namespace -> inspect pod details and logs
```
CLI fallback when AKS-MCP cannot perform the Kubernetes baseline read — the same **[`aks-baseline`](../../../scripts/aks-baseline.sh)** script also covers node readiness, unhealthy pods, kube-system health, and recent warning events. Pass `--namespace` to include an affected namespace, then deep-dive on a specific pod:
```bash
kubectl cluster-info
kubectl get nodes -o wide
kubectl get pods -n kube-system
kubectl get events -A --sort-by=.lastTimestamp
kubectl get pods -n <namespace>
```
For pod detail and logs, gather the read-only evidence bundle (describe, current + previous logs, resources vs usage) with the pod-evidence script — [`../../../scripts/pod-evidence.sh`](../../../scripts/pod-evidence.sh) / [`../../../scripts/pod-evidence.ps1`](../../../scripts/pod-evidence.ps1):
```bash
../../../scripts/pod-evidence.sh <pod-name> -n <namespace>
../../../scripts/pod-evidence.sh --all-failing
kubectl describe pod <pod-name> -n <namespace>
kubectl logs <pod-name> -n <namespace> --previous
```
```powershell
../../../scripts/pod-evidence.ps1 <pod-name> -Namespace <namespace>
../../../scripts/pod-evidence.ps1 -AllFailing
```
```powershell
# PowerShell
.\scripts\aks-baseline.ps1 -ResourceGroup <resource-group> -Cluster <cluster-name> -Namespace <namespace>
```
## Connectivity Flow
```text
pod -> service -> endpoints -> ingress or load balancer -> DNS -> network controls
```
CLI fallback when AKS-MCP cannot perform the connectivity read:
```bash
kubectl get pods -n <namespace> -o wide
kubectl get svc -n <namespace>
kubectl get endpoints -n <namespace>
kubectl get ingress -n <namespace>
kubectl describe ingress <ingress-name> -n <namespace>
```
## Detector Flow
```text
resolve cluster resource ID -> list detectors or choose category -> select a focused time window -> run the detector or category -> rank critical findings above warnings -> ignore emerging issues when choosing the primary root cause
```
## Monitoring Flow
```text
check resource health -> inspect metrics -> verify diagnostics settings -> inspect control plane logs if available -> correlate with Application Insights or namespace symptoms
```
## Scheduling Flow
```text
pod events -> node capacity -> taints and tolerations -> affinity rules -> PVC state -> quotas
```
CLI fallback when AKS-MCP cannot perform the scheduling read:
```bash
kubectl describe pod <pod-name> -n <namespace>
kubectl get nodes -o wide
kubectl describe node <node-name>
kubectl get pvc -n <namespace>
kubectl describe quota -n <namespace>
```
## Deep Diagnostics Flow (Inspektor Gadget)
```text
Standard diagnostics inconclusive -> select gadget from symptom-to-gadget map -> run `scripts/run-ig.sh` (or `run-ig.ps1`; resolves node, applies timeout) -> interpret output -> correlate with prior evidence
```
Use when steps 1–3 of the evidence order (Azure-side, Kubernetes-side, and detector evidence) do not reveal root cause. See [inspektor-gadget.md](inspektor-gadget.md) for the full gadget catalog and command patterns.
## Safety Boundary
Treat the following as change operations and avoid them unless the user explicitly asks for remediation:
- deleting or restarting pods
- cordon and drain operations
- scaling workloads or node pools
- cluster upgrade operations
- DNS, route, NSG, or firewall changes
inspektor-gadget.md 7.6 KB
# Inspektor Gadget (IG) Reference
Use Inspektor Gadget for low-level node/pod diagnostics when `kubectl` is insufficient.
## Run Script
Invoke gadgets with the `run-ig` script ([`scripts/run-ig.sh`](../../../scripts/run-ig.sh) /
[`scripts/run-ig.ps1`](../../../scripts/run-ig.ps1)). It resolves the node from the pod, injects
the pinned IG image/version, applies the default `--timeout` for the gadget type, adds the k8s
filters, and handles the `tcpdump` variant. You still choose **which** gadget (see the
Symptom-to-Gadget Map) and interpret the output.
```bash
./scripts/run-ig.sh --gadget trace_dns --pod <pod> --ns <ns> # node auto-resolved
./scripts/run-ig.sh --gadget snapshot_process --node <node> # node-wide
./scripts/run-ig.sh --gadget trace_dns --pod <pod> --ns <ns> --dry-run
```
```powershell
.\scripts\run-ig.ps1 -Gadget trace_dns -Pod <pod> -Namespace <ns>
```
**Options** (bash flags below; PowerShell uses PascalCase equivalents: `-Gadget`, `-Pod`,
`-Namespace`/`-Ns`, `-Node`, `-Container`, `-Timeout`, `-Filter`, `-Pf`, `-IgVersion`,
`-DryRun`): `--gadget` (required); target `--pod`/`--ns` **or** `--node`; `--container`;
`--timeout <s>` override; `--filter <arg>` (repeatable IG-flag passthrough, e.g.
`--filter --max-entries --filter 20`); `--pf "<expr>"` (tcpdump only); `--ig-version <tag>`;
`--dry-run`. Default timeout by gadget name: `snapshot_*`/`top_*` → 5s,
`trace_*`/`profile_*`/`tcpdump` → 30s. Returns the gadget JSON (pcap-ng for tcpdump) plus a
`Ran gadget X on node Y` summary. IG version is pinned to `v0.51.0` in the scripts.
> **Approval required:** IG uses `kubectl debug --profile=sysadmin` (a privileged debug pod).
> **Ask the user before running the script** and confirm RBAC; use `--dry-run` to preview.
## Common Filters
The k8s scope filters and `--timeout` are set by the script. Pass any other IG flag below via
its repeatable `--filter`, e.g. `--filter --max-entries --filter 20`.
| Filter | Description |
|---|---|
| `--max-entries <n>` | Max entries per batch for top/profile gadgets |
| `--map-fetch-interval <dur>` | Map fetch interval for top (except `top_process`) and profile gadgets (default `1000ms`) |
| `--interval <dur>` | Reporting interval for `top_process` only (e.g. `5s`) |
| `--syscall-filters <list>` | Comma-separated syscalls for `traceloop` (e.g. `open,connect,accept`). **Always specify** to limit data volume |
> **Tip:** For top/profile, keep `--map-fetch-interval` ≤ half of `--timeout` to collect ≥1 batch. `top_process` uses `--interval` instead of `--map-fetch-interval`.
## Gadget Catalog
### Networking
| Gadget | Type | What It Does | When To Use |
|---|---|---|---|
| `trace_dns` | trace | Trace DNS queries and responses with latency | DNS failures, NXDOMAIN, SERVFAIL, slow resolution, intermittent DNS |
| `trace_tcp` | trace | Trace TCP connect/accept/close events | Connection refused, timeouts, unexpected drops, mapping pod connectivity |
| `trace_tcpretrans` | trace | Trace TCP retransmissions | Network congestion, lossy links, high latency between pods/services |
| `trace_bind` | trace | Trace socket bind calls | Port conflicts, address-already-in-use errors |
| `trace_sni` | trace | Trace TLS SNI (Server Name Indication) values | HTTPS routing issues, ingress TLS debugging, mTLS problems |
| `snapshot_socket` | snapshot | List open sockets (TCP/UDP/Unix) | Port conflicts, listening ports, connection leaks, ECONNREFUSED |
| `tcpdump` | special | Capture raw packets in pcap-ng format | Deep packet inspection, protocol-level debugging, reproducing network issues |
#### tcpdump gadget
Run via `--gadget tcpdump`; the script sets `-o pcap-ng` and pipes to `tcpdump -nvr -` when
available. Use `--pf "<expr>"` for tcpdump filters (e.g., `port 80`, `host 10.0.0.1`); `--pf`
is only valid for the `tcpdump` gadget.
```bash
./scripts/run-ig.sh --gadget tcpdump --pod <pod> --ns <ns> --pf "port 80"
```
### Process & Workload
| Gadget | Type | What It Does | When To Use |
|---|---|---|---|
| `snapshot_process` | snapshot | List running processes in pod/node | PID pressure, unknown processes, verifying entrypoint, CrashLoopBackOff |
| `trace_exec` | trace | Trace process execution (execve calls) | CrashLoopBackOff (what actually runs), unexpected child processes, security audit |
| `trace_oomkill` | trace | Trace OOM kill events with victim details | OOMKilled pods — see which process was killed, memory usage at kill time |
| `trace_signal` | trace | Trace signals delivered to processes | Unexpected SIGKILL/SIGTERM, liveness probe kills, graceful shutdown issues |
| `top_process` | top | Rank processes by CPU/memory usage | Identifying resource-hungry processes inside a pod or across a node |
| `profile_cpu` | profile | CPU profiling via stack sampling | High CPU usage investigation, finding hot code paths |
| `traceloop` | trace | Record syscalls as a flight recorder | Catch-all for intermittent issues. **Always use `--syscall-filters`** (e.g., `open,connect,accept`) to limit data volume |
### File & Storage
| Gadget | Type | What It Does | When To Use |
|---|---|---|---|
| `trace_open` | trace | Trace openat syscall | Missing config/secret files (ENOENT), permission denied (EACCES), startup failures |
| `trace_fsslower` | trace | Trace slow filesystem operations | Slow disk I/O, PVC performance issues, NFS/Azure Disk latency |
| `top_file` | top | Rank files by read/write activity | Identifying I/O-heavy files, noisy log writers, disk pressure diagnosis |
### Security & Audit
| Gadget | Type | What It Does | When To Use |
|---|---|---|---|
| `trace_capabilities` | trace | Trace Linux capability checks | Permission denied from dropped capabilities, SecurityContext debugging |
## Symptom-to-Gadget Map
| Symptom | Gadget(s) |
|---|---|
| DNS resolution failures | `trace_dns` |
| Connection refused / timeout | `trace_tcp` + `snapshot_socket` |
| Silent connection drops | `trace_tcpretrans` |
| High network latency | `trace_tcpretrans` |
| TLS / HTTPS routing issues | `trace_sni` |
| Port already in use | `trace_bind` + `snapshot_socket` |
| CrashLoopBackOff (unknown cause) | `trace_exec` + `trace_open` |
| OOMKilled pods | `trace_oomkill` + `top_process` |
| Pod killed unexpectedly | `trace_signal` |
| PID pressure on node | `snapshot_process` + `top_process` |
| "Too many open files" | `top_file` |
| Missing config / secret mount | `trace_open` |
| Slow disk / PVC performance | `trace_fsslower` + `top_file` |
| Permission denied (capabilities) | `trace_capabilities` |
| High CPU (unknown cause) | `profile_cpu` + `top_process` |
| Deep packet inspection | `tcpdump` |
| Catch-all / intermittent issues | `traceloop` (use `--syscall-filters`) |
## Gadget Type Reference
| Type | Behavior | IG --timeout |
|---|---|---|
| `snapshot` | Point-in-time data, returns immediately | `--timeout 5` |
| `top` | Aggregated view, returns quickly | `--timeout 5` |
| `trace` | Streams events in real-time | `--timeout 30` |
| `profile` | Samples over a duration | `--timeout 30` |
| `tcpdump` | Streams pcap-ng data, pipe to `tcpdump -nvr -` | `--timeout 30` |
## Guardrails
- IG gadgets are **read-only** — they do not modify cluster or application state.
- Invoke gadgets through `run-ig` (`scripts/run-ig.sh` / `scripts/run-ig.ps1`); it resolves the node and applies the correct timeout. **Ask the user before running it** (privileged debug pod).
- The script picks the default `--timeout` by gadget type. Prefer snapshot/top for quick checks; trace/profile for behavior over time. Override with `--timeout` when needed.
- For reproduction: launch a trace gadget first, then reproduce the problem. The debug pod persists after the gadget exits, so run `kubectl logs <debug-pod>` to retrieve the captured output afterward.
structured-input-modes.md 1.5 KB
# AKS Structured Input Modes
Use this reference when the troubleshooting request already contains structured inputs.
## Detector-backed Mode
Use when AKS-aware detectors or AppLens-style insights are available.
Decision rules:
- Ignore findings where the detector is `emergingIssues`.
- Prefer critical findings over warnings.
- Prefer findings with more concrete remediation detail when choosing the likely root problem.
- Preserve per-insight output: problem summary, root-problem flag, affected resources, suggested commands.
## Warning Events Mode
Use when the request includes Kubernetes warning events.
Expected output:
- summary of the events and their impact
- likely cause or causes
- next kubectl checks
- monitoring follow-up
## Metrics Scan Mode
Use when the request includes CPU or memory time-series data.
Expected output:
- healthy or unhealthy status
- anomaly timestamps and explanations
- suggestion tied to the observed metric pressure
## Generic Symptoms Mode
Use when the request includes resource symptoms but not detector results, warning events, or time-series metrics.
Expected output:
- symptom summary by resource
- likely failure domain
- next evidence-collection steps
## Learn Grounding Fallback
If the first troubleshooting pass is incomplete, search Microsoft Learn using:
- the user prompt
- the parsed problem names
- the AKS troubleshooting context
Use Learn grounding to refine or validate the root-cause hypothesis, not to replace observed evidence.
troubleshooting/compute/references/
cannot-connect-to-vm.md 3.1 KB
# Cannot Connect to VM
Router for Azure VM connectivity issues. Determine OS first: Windows usually means RDP/3389 and TermService; Linux means SSH/22 and sshd; other images usually use SSH but may need current docs.
## Routing
| Signal | Category | Reference |
| --- | --- | --- |
| can't RDP, timeout, black screen, internal error | RDP | [rdp-connectivity.md](rdp-connectivity.md) |
| can't SSH, refused, permission denied, publickey | SSH | [ssh-connectivity.md](ssh-connectivity.md) |
| NSG, public IP, NIC, routes, effective rules | Network | [network-connectivity.md](network-connectivity.md) |
| Windows Firewall, iptables, firewalld, UFW | Guest firewall | [firewall-blocking.md](firewall-blocking.md) |
| VM agent, Run Command timeout, Serial Console, boot diagnostics | VM agent/tools | [vm-agent-not-responding.md](vm-agent-not-responding.md) |
| password, credentials, access denied, CredSSP, account expired | Credential/auth | [credential-auth-errors.md](credential-auth-errors.md) |
| TermService, RDP disabled, changed port, TLS cert, NLA, licensing | RDP service/config | [rdp-service-config.md](rdp-service-config.md) |
## Workflow
1. Pick the symptom category and open the linked reference.
2. Match the user's symptom to a solution row and fetch current docs for that row.
3. Before extension-backed operations, run pre-flight checks.
4. Return evidence, likely cause, safe next command, remediation, and escalation.
## Pre-Flight Safety Checks
Run before `az vm user update`, `az vm user reset-ssh`, `az vm user reset-remote-desktop`, `az vm run-command invoke`, or any extension-backed operation.
```bash
az vm get-instance-view --name <vm> -g <rg> \
--query "instanceView.{power:[statuses[?starts_with(code,'PowerState/')]][0][0].code,prov:[statuses[?starts_with(code,'ProvisioningState/')]][0][0].code,agent:vmAgent.statuses[0].displayStatus}" -o json
az vm extension list --vm-name <vm> -g <rg> \
--query "[].{name:name,state:provisioningState}" -o table
```
| Check | Safe | Unsafe |
| --- | --- | --- |
| Power | `PowerState/running` | other, missing, or query error |
| Provisioning | `ProvisioningState/succeeded` | creating/updating/deleting/failed, missing, or query error |
| VM agent | `Ready` | not ready, null, missing, or query error |
| Extensions | all `Succeeded` or none | creating/updating/deleting/failed |
If unsafe: stop, name the failed check, and use non-agent options such as Serial Console, offline repair VM, or Portal actions. If transient, wait and rerun checks only.
## Escalation
Check Resource Health, then offer restart or redeploy only with user approval. For broad docs, use [Windows RDP], [Linux SSH], [Windows VM hub], or [Linux VM hub].
[Windows RDP]: https://learn.microsoft.com/en-us/troubleshoot/azure/virtual-machines/windows/troubleshoot-rdp-connection
[Linux SSH]: https://learn.microsoft.com/en-us/troubleshoot/azure/virtual-machines/linux/troubleshoot-ssh-connection
[Windows VM hub]: https://learn.microsoft.com/en-us/troubleshoot/azure/virtual-machines/windows/welcome-virtual-machines-windows
[Linux VM hub]: https://learn.microsoft.com/en-us/troubleshoot/azure/virtual-machines/linux/welcome-virtual-machines-linuxcredential-auth-errors.md 2.3 KB
# Credential and Authentication Errors
Use when the VM is reachable but sign-in fails.
## Symptoms -> Solutions
| Symptom | OS | Action | Docs |
| --- | --- | --- | --- |
| credentials failed, must change password, account expired | Windows | Reset password; extend account if needed | [Reset RDP] |
| trust relationship failed | Windows | Reset machine account or rejoin domain | [RDP overview] |
| access denied, connection denied, wrong local/domain format | Windows | Add Remote Desktop Users; use `VMNAME\user` or `DOMAIN\user` | [RDP errors] |
| CredSSP encryption oracle | Windows | Temporary client workaround; patch both sides | [CredSSP] |
| permission denied publickey/password | Linux | Verify user/key/password auth; reset key/password if needed | [SSH detail] |
| locked account | Linux | Unlock via Run Command or Serial Console | [SSH overview] |
| Entra ID SSH denied | Linux | Assign VM Admin/User Login role | [SSH overview] |
| sudo prompt fails | Linux | Fix sudoers via Run Command or Serial Console | [SSH overview] |
## Quick Commands
> Commands use VM agent/extensions. Run [Pre-Flight Safety Checks](cannot-connect-to-vm.md#pre-flight-safety-checks) first.
```bash
# Windows password / RDP reset
az vm user update --name <vm> -g <rg> -u <user> -p '<new-password>'
az vm user reset-remote-desktop --name <vm> -g <rg>
# Linux key/password reset or unlock
az vm user update --name <vm> -g <rg> -u <user> --ssh-key-value "<ssh-public-key>"
az vm user update --name <vm> -g <rg> -u <user> -p '<new-password>'
az vm run-command invoke --name <vm> -g <rg> --command-id RunShellScript --scripts "passwd -u <user>"
```
[Reset RDP]: https://learn.microsoft.com/en-us/troubleshoot/azure/virtual-machines/windows/reset-rdp
[RDP overview]: https://learn.microsoft.com/en-us/troubleshoot/azure/virtual-machines/windows/troubleshoot-rdp-connection
[RDP errors]: https://learn.microsoft.com/en-us/troubleshoot/azure/virtual-machines/windows/troubleshoot-specific-rdp-errors
[CredSSP]: https://learn.microsoft.com/en-us/troubleshoot/azure/virtual-machines/windows/credssp-encryption-oracle-remediation
[SSH detail]: https://learn.microsoft.com/en-us/troubleshoot/azure/virtual-machines/linux/detailed-troubleshoot-ssh-connection
[SSH overview]: https://learn.microsoft.com/en-us/troubleshoot/azure/virtual-machines/linux/troubleshoot-ssh-connectionfirewall-blocking.md 2.3 KB
# Firewall Blocking Connectivity
Use when NSG/platform rules allow traffic but the guest OS firewall blocks RDP/SSH.
## Symptoms -> Solutions
| Symptom | OS | Action | Docs |
| --- | --- | --- | --- |
| Windows Firewall blocks RDP | Windows | Enable Remote Desktop firewall group | [Guest firewall] |
| BlockInboundAlways or bad policy | Windows | Reset to `blockinbound,allowoutbound` | [Firewall rule] |
| third-party AV/firewall | Windows | Stop for test, then reconfigure | [Guest firewall] |
| iptables/nftables blocks SSH | Linux | Insert allow rule or remove blocking chain | [SSH overview] |
| firewalld blocks SSH | Linux | Open SSH service in active zone | [SSH overview] |
| UFW blocks SSH | Linux | `ufw allow 22/tcp` or disable temporarily | [SSH overview] |
| no guest access | Any | Use Serial Console or offline repair | [Offline firewall] / [Linux repair] |
## Quick Commands
> Commands use VM agent/extensions. Run [Pre-Flight Safety Checks](cannot-connect-to-vm.md#pre-flight-safety-checks) first.
```bash
# Windows
az vm user reset-remote-desktop --name <vm> -g <rg>
az vm run-command invoke --name <vm> -g <rg> --command-id RunPowerShellScript \
--scripts "netsh advfirewall firewall set rule group='Remote Desktop' new enable=yes"
# Linux
az vm run-command invoke --name <vm> -g <rg> --command-id RunShellScript \
--scripts "iptables -L -n; iptables -I INPUT -p tcp --dport 22 -j ACCEPT"
az vm run-command invoke --name <vm> -g <rg> --command-id RunShellScript \
--scripts "firewall-cmd --add-service=ssh --permanent && firewall-cmd --reload"
az vm run-command invoke --name <vm> -g <rg> --command-id RunShellScript \
--scripts "ufw status; ufw allow 22/tcp"
```
[Guest firewall]: https://learn.microsoft.com/en-us/troubleshoot/azure/virtual-machines/windows/guest-os-firewall-blocking-inbound-traffic
[Firewall rule]: https://learn.microsoft.com/en-us/troubleshoot/azure/virtual-machines/windows/enable-disable-firewall-rule-guest-os
[Offline firewall]: https://learn.microsoft.com/en-us/troubleshoot/azure/virtual-machines/windows/disable-guest-os-firewall-windows
[SSH overview]: https://learn.microsoft.com/en-us/troubleshoot/azure/virtual-machines/linux/troubleshoot-ssh-connection
[Linux repair]: https://learn.microsoft.com/en-us/troubleshoot/azure/virtual-machines/linux/repair-linux-vm-using-azure-virtual-machine-repair-commandsnetwork-connectivity.md 2.5 KB
# Network Connectivity Problems
Use when the VM is running but unreachable due to Azure network configuration: NSG, routing, NIC, public IP, or DNS.
## Symptoms -> Solutions
| Symptom | OS | Action | Docs |
| --- | --- | --- | --- |
| no NSG allow for RDP/SSH | Any | Add inbound TCP 3389/22 from approved source | [RDP NSG] |
| NIC and subnet NSGs conflict | Any | Traffic must pass both; inspect effective rules | [Traffic filter] |
| UDR sends traffic to NVA | Any | Check effective routes and NVA forwarding | [Routing] |
| no public IP | Any | Add public IP or use Bastion/private path | [Public IP] |
| guest NIC disabled/down | Windows/Linux | Enable NIC via Run Command or Serial Console | [RDP NIC] / [SSH overview] |
| static guest IP misconfig | Windows/Linux | Restore DHCP guest config | [Reset NIC] / [SSH overview] |
| ghost NIC after disk swap/resize | Windows | Reset network interface | [Reset NIC] |
| DNS failure | Any | Check DNS; Azure default is `168.63.129.16` | [DHCP] |
## Quick Commands
```bash
az network nic list-effective-nsg --name <nic> -g <rg>
az network nic show-effective-route-table --name <nic> -g <rg> -o table
az vm list-ip-addresses --name <vm> -g <rg> -o table
az network watcher test-connectivity --source-resource <vm-resource-id> \
--dest-address <ip> --dest-port <port> -g <rg>
```
> Linux guest NIC commands use VM agent/extensions. Run [Pre-Flight Safety Checks](cannot-connect-to-vm.md#pre-flight-safety-checks) first.
```bash
az vm repair reset-nic --name <vm> -g <rg> --yes
az vm run-command invoke --name <vm> -g <rg> --command-id RunShellScript \
--scripts "ip link show; ip addr show; ip link set eth0 up && dhclient eth0"
```
[RDP NSG]: https://learn.microsoft.com/en-us/troubleshoot/azure/virtual-machines/windows/troubleshoot-rdp-nsg-problem
[Traffic filter]: https://learn.microsoft.com/en-us/azure/network-watcher/diagnose-vm-network-traffic-filtering-problem
[Routing]: https://learn.microsoft.com/en-us/azure/network-watcher/diagnose-vm-network-routing-problem
[Public IP]: https://learn.microsoft.com/en-us/azure/virtual-network/ip-services/public-ip-addresses
[RDP NIC]: https://learn.microsoft.com/en-us/troubleshoot/azure/virtual-machines/windows/troubleshoot-rdp-nic-disabled
[Reset NIC]: https://learn.microsoft.com/en-us/troubleshoot/azure/virtual-machines/windows/reset-network-interface
[DHCP]: https://learn.microsoft.com/en-us/troubleshoot/azure/virtual-machines/windows/troubleshoot-rdp-dhcp-disabled
[SSH overview]: https://learn.microsoft.com/en-us/troubleshoot/azure/virtual-machines/linux/troubleshoot-ssh-connectionrdp-connectivity.md 2.7 KB
# Unable to RDP into the VM
Use for Windows VM RDP timeouts, refused connections, black screen, or RDP error dialogs.
## Symptoms -> Solutions
| Symptom | Action | Docs |
| --- | --- | --- |
| timeout/no response | Check power state, public IP, NSG allow for 3389 | [RDP NSG] |
| timeout with NSG OK | Check guest firewall | [Guest firewall] |
| credentials failed | Reset password or username format | [RDP errors] |
| internal/security/authentication error | Check TLS, NLA, CredSSP, certificate, clock skew | [Internal] / [General] |
| black screen after login | Check Explorer, GPU driver, GPO, session state | [Detailed] |
| license server unavailable | Fix or remove RDS licensing role | [RDP errors] |
| cannot find computer | Check public IP, DNS, and VM allocation | [RDP errors] |
| connects then disconnects | Check session limits, idle timeout, resources | [RDP overview] |
| works from some IPs | Check NSG source restriction | [RDP NSG] |
| Event IDs in logs | Match event ID to documented cause | [Event IDs] |
| guest NIC disabled | Enable NIC via safe command path | [RDP NIC] |
## Quick Commands
> Commands marked by reset/update use VM agent/extensions. Run [Pre-Flight Safety Checks](cannot-connect-to-vm.md#pre-flight-safety-checks) first.
```bash
az vm get-instance-view --name <vm> -g <rg> --query "instanceView.statuses" -o table
az network nsg rule list --nsg-name <nsg> -g <rg> -o table
az network watcher test-ip-flow --direction Inbound --protocol TCP \
--local <vm-private-ip>:3389 --remote <your-public-ip>:* --vm <vm> -g <rg>
az vm user reset-remote-desktop --name <vm> -g <rg>
az vm user update --name <vm> -g <rg> -u <user> -p '<new-password>'
```
[RDP NSG]: https://learn.microsoft.com/en-us/troubleshoot/azure/virtual-machines/windows/troubleshoot-rdp-nsg-problem
[Guest firewall]: https://learn.microsoft.com/en-us/troubleshoot/azure/virtual-machines/windows/guest-os-firewall-blocking-inbound-traffic
[RDP errors]: https://learn.microsoft.com/en-us/troubleshoot/azure/virtual-machines/windows/troubleshoot-specific-rdp-errors
[Internal]: https://learn.microsoft.com/en-us/troubleshoot/azure/virtual-machines/windows/troubleshoot-rdp-internal-error
[General]: https://learn.microsoft.com/en-us/troubleshoot/azure/virtual-machines/windows/troubleshoot-rdp-general-error
[Detailed]: https://learn.microsoft.com/en-us/troubleshoot/azure/virtual-machines/windows/detailed-troubleshoot-rdp
[RDP overview]: https://learn.microsoft.com/en-us/troubleshoot/azure/virtual-machines/windows/troubleshoot-rdp-connection
[Event IDs]: https://learn.microsoft.com/en-us/troubleshoot/azure/virtual-machines/windows/event-id-troubleshoot-vm-rdp-connecton
[RDP NIC]: https://learn.microsoft.com/en-us/troubleshoot/azure/virtual-machines/windows/troubleshoot-rdp-nic-disabledrdp-service-config.md 3.2 KB
# RDP Service and Configuration Issues
VM is reachable but the RDP service itself is broken or misconfigured.
## Symptoms → Solutions
| Symptom | Solution | Documentation |
| -------------------------------------- | ---------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ |
| TermService not running | Start the service and set to Automatic | [Reset RDP service](https://learn.microsoft.com/en-us/troubleshoot/azure/virtual-machines/windows/reset-rdp) |
| RDP port changed from 3389 | Reset port or update NSG to allow the custom port | [Detailed RDP troubleshooting](https://learn.microsoft.com/en-us/troubleshoot/azure/virtual-machines/windows/detailed-troubleshoot-rdp) |
| RDP disabled (fDenyTSConnections = 1) | Reset RDP config via CLI or Portal | [Reset RDP service](https://learn.microsoft.com/en-us/troubleshoot/azure/virtual-machines/windows/reset-rdp) |
| TLS/SSL certificate expired or corrupt | Delete cert and restart TermService to regenerate | [RDP internal error](https://learn.microsoft.com/en-us/troubleshoot/azure/virtual-machines/windows/troubleshoot-rdp-internal-error) |
| NLA/Security Layer mismatch | Temporarily disable NLA for recovery | [RDP general error](https://learn.microsoft.com/en-us/troubleshoot/azure/virtual-machines/windows/troubleshoot-rdp-general-error) |
| GPO overriding local RDP settings | Check `HKLM:\SOFTWARE\Policies\Microsoft\Windows NT\Terminal Services` | [Detailed RDP troubleshooting](https://learn.microsoft.com/en-us/troubleshoot/azure/virtual-machines/windows/detailed-troubleshoot-rdp) |
| RDS licensing expired | Remove RDSH role or configure license server | [Specific RDP errors](https://learn.microsoft.com/en-us/troubleshoot/azure/virtual-machines/windows/troubleshoot-specific-rdp-errors#rdplicense) |
## Quick Commands
> ⚠️ **Warning:** Commands marked with ⚡ use the VM agent/extensions. Run [Pre-Flight Safety Checks](cannot-connect-to-vm.md#pre-flight-safety-checks) before using them.
```bash
# ⚡ Reset all RDP configuration to defaults
az vm user reset-remote-desktop --name <vm-name> -g <resource-group>
# ⚡ Check TermService status via Run Command
az vm run-command invoke --name <vm-name> -g <resource-group> \
--command-id RunPowerShellScript --scripts "Get-Service TermService | Select-Object Status, StartType"
# Restart VM (if RDP service is unrecoverable — requires user approval)
az vm restart --name <vm-name> -g <resource-group>
# Redeploy VM (moves to new host — last resort, requires user approval)
az vm redeploy --name <vm-name> -g <resource-group>
```ssh-connectivity.md 2.1 KB
# Unable to SSH into the VM
Use for Linux VM SSH failures.
## Symptoms -> Solutions
| Symptom | Action | Docs |
| --- | --- | --- |
| connection refused on 22 | Check sshd running/listening and port config | [SSH overview] |
| connection timed out | Check power state, public IP, NSG, routes | [SSH overview] |
| permission denied publickey/password | Verify user, key, password auth; reset key/password | [SSH detail] |
| host key verification failed | Remove stale `known_hosts` entry | [SSH detail] |
| server closed connection | Check disk, PAM, sshd config | [SSH detail] |
| hangs with no response | Check firewall, routes, NIC | [SSH overview] |
| Debian-specific failure | Check Debian networking/sshd doc | [Debian] |
| SELinux blocks sshd | Fix SELinux policy or temporarily permissive | [SELinux] |
| Entra ID SSH denied | Assign VM Admin/User Login role | [SSH overview] |
| VM not booting/UEFI failure | Use boot diagnostics and repair VM | [UEFI] |
## Quick Commands
> Commands use VM agent/extensions. Run [Pre-Flight Safety Checks](cannot-connect-to-vm.md#pre-flight-safety-checks) first.
```bash
az vm user reset-ssh --name <vm> -g <rg>
az vm user update --name <vm> -g <rg> -u <user> --ssh-key-value "<ssh-public-key>"
az vm user update --name <vm> -g <rg> -u <user> -p '<new-password>'
az vm run-command invoke --name <vm> -g <rg> --command-id RunShellScript \
--scripts "systemctl status sshd; getenforce"
az vm run-command invoke --name <vm> -g <rg> --command-id RunShellScript \
--scripts "setenforce 0"
```
[SSH overview]: https://learn.microsoft.com/en-us/troubleshoot/azure/virtual-machines/linux/troubleshoot-ssh-connection
[SSH detail]: https://learn.microsoft.com/en-us/troubleshoot/azure/virtual-machines/linux/detailed-troubleshoot-ssh-connection
[Debian]: https://learn.microsoft.com/en-us/troubleshoot/azure/virtual-machines/linux/cannot-connect-debian-linux
[SELinux]: https://learn.microsoft.com/en-us/troubleshoot/azure/virtual-machines/linux/linux-selinux-troubleshooting
[UEFI]: https://learn.microsoft.com/en-us/troubleshoot/azure/virtual-machines/linux/azure-linux-vm-uefi-boot-failuresvm-agent-not-responding.md 3.6 KB
# VM Agent Not Responding
Run Command and password reset depend on the VM agent. If the agent is unhealthy, alternative methods are needed.
> ⚠️ **OS Note:** Serial Console, Boot Diagnostics, and repair VM commands are available for both Windows and Linux but use separate doc pages and tools. Match the correct OS below.
## Symptoms → Solutions
| Symptom | OS | Solution | Documentation |
| ---------------------------------------------------------------- | ------- | -------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Run Command times out | Windows | VM agent may be down — use Serial Console instead | [Serial Console — Windows](https://learn.microsoft.com/en-us/troubleshoot/azure/virtual-machines/windows/serial-console-overview) |
| Run Command times out | Linux | VM agent may be down — use Serial Console instead | [Serial Console — Linux](https://learn.microsoft.com/en-us/troubleshoot/azure/virtual-machines/linux/serial-console-linux) |
| Password reset fails via Portal/CLI | Windows | VMAccess extension can't communicate — use offline reset | [Reset password without agent](https://learn.microsoft.com/en-us/troubleshoot/azure/virtual-machines/windows/reset-local-password-without-agent) |
| Password/key reset fails via Portal/CLI | Linux | VMAccess extension can't communicate — use Serial Console | [Serial Console — Linux](https://learn.microsoft.com/en-us/troubleshoot/azure/virtual-machines/linux/serial-console-linux) |
| VM not booting (Boot Diagnostics shows BSOD/stuck) | Windows | OS-level issue — use repair VM for offline fix | [Repair Windows VM](https://learn.microsoft.com/en-us/troubleshoot/azure/virtual-machines/windows/repair-windows-vm-using-azure-virtual-machine-repair-commands) |
| VM not booting (Boot Diagnostics shows kernel panic/stuck) | Linux | Use repair VM for offline Linux disk fix | [Repair Linux VM](https://learn.microsoft.com/en-us/troubleshoot/azure/virtual-machines/linux/repair-linux-vm-using-azure-virtual-machine-repair-commands) |
| VMAccess extension error on domain controller | Windows | VMAccess doesn't support DCs — use Serial Console | [Serial Console — Windows](https://learn.microsoft.com/en-us/troubleshoot/azure/virtual-machines/windows/serial-console-overview) |
## Quick Commands
```bash
# Connect to Serial Console via CLI
az serial-console connect --name <vm-name> -g <resource-group>
# Enable boot diagnostics (required for Serial Console)
az vm boot-diagnostics enable --name <vm-name> -g <resource-group>
# Get boot diagnostics screenshot/log
az vm boot-diagnostics get-boot-log --name <vm-name> -g <resource-group>
# Create repair VM for offline fixes
az vm repair create --name <vm-name> -g <resource-group> \
--repair-username repairadmin --repair-password '<password>'
# Restore after offline fix
az vm repair restore --name <vm-name> -g <resource-group>
``` troubleshooting/compute/
vm-troubleshooting.md 3.3 KB
# Azure VM Connectivity Troubleshooting
Primary compute troubleshooting guide for incidents routed from [../../SKILL.md](../../SKILL.md). Use for Azure VM RDP/SSH failures, NSG/firewall blocks, credential resets, and VM agent/tooling issues.
## Quick Reference
| Property | Details |
| --- | --- |
| Best for | RDP/SSH failures, port 3389/22 timeouts, black screen, credential reset, VM agent issues |
| Primary tools | `mcp_azure_mcp_compute`, `mcp_azure_mcp_resourcehealth`, `mcp_azure_mcp_monitor`, CLI fallback |
| Router | [references/cannot-connect-to-vm.md](references/cannot-connect-to-vm.md) |
## MCP Tools
| Tool | Purpose |
| --- | --- |
| `mcp_azure_mcp_compute` | VM state, instance view, VM agent, extension, NIC evidence |
| `mcp_azure_mcp_resourcehealth` | Platform health before deeper diagnosis |
| `mcp_azure_mcp_monitor` | Logs/metrics when workspace or resource scope is known |
| `mcp_azure_mcp_documentation` | Current Microsoft Learn guidance for the matched symptom |
## When to Use
- "can't connect to my VM", "can't RDP", "can't SSH"
- RDP/SSH timeout, refused, black screen, internal error, session drop
- reset VM password, wrong credentials, access denied
- NSG, guest firewall, port 3389/22, public IP, NIC, Serial Console, Run Command, VM agent
## Guardrails
- Default to read-only diagnostics; quote evidence before concluding root cause.
- Do not run extension-backed commands (`az vm user update`, `az vm user reset-ssh`, `az vm user reset-remote-desktop`, `az vm run-command invoke`) until [Pre-Flight Safety Checks](references/cannot-connect-to-vm.md#pre-flight-safety-checks) pass.
- Do not restart, redeploy, deallocate, or delete unless the user explicitly approves remediation.
- If multiple issues appear, fix network-layer blockers before agent-dependent fixes.
## Evidence Order
1. VM state: power, provisioning, VM agent, extension states.
2. Network layer: public IP, NIC/subnet NSGs, effective routes, IP flow.
3. Guest OS: services and firewall via Run Command only when agent checks are safe.
## Workflow
1. Classify intent. If unclear, ask whether the user uses RDP or SSH and what error appears.
2. Open [references/cannot-connect-to-vm.md](references/cannot-connect-to-vm.md), choose the matching symptom category, then open that reference.
3. If a command uses the VM agent/extensions, run pre-flight checks first and stop on any unsafe result.
4. Use `mcp_azure_mcp_documentation` to fetch current docs for the selected URL or symptom.
5. Respond with evidence, likely cause, safe diagnostic/fix commands, and escalation path.
```yaml
mcp_azure_mcp_documentation
intent: "find current Azure VM RDP SSH troubleshooting docs for <symptom>"
parameters:
query: "<documentation URL or user's symptom>"
```
## Error Handling
| Error | Action |
| --- | --- |
| Docs lookup empty | Use the reference quick commands and tell user the doc URL may have changed |
| VM name/resource group wrong | Ask user to verify resource identity |
| Run Command timeout or `VMAgentStatusCommunicationError` | Do not run extension commands; use Serial Console/offline repair |
| Serial Console unavailable | Enable Boot Diagnostics first |
| Password reset fails | Check VMAccess alternatives in the credential and VM agent references |
| Extension stuck updating | Do not add extensions; use Portal/Serial Console/offline repair | troubleshooting/messaging/
README.md 1.6 KB
# Azure Messaging Troubleshooting
Diagnose and resolve issues with Azure Event Hubs and Service Bus SDKs.
## Routing
| Symptom | Guide |
|---------|-------|
| Connection failures, firewall, IP/VNet, WebSocket | [service-troubleshooting.md](service-troubleshooting.md) |
| SDK-specific errors (see language below) | Language guide |
## SDK Troubleshooting by Language
- **Event Hubs**: [Python](azure-eventhubs-py.md) | [Java](azure-eventhubs-java.md) | [JS](azure-eventhubs-js.md) | [.NET](azure-eventhubs-dotnet.md)
- **Service Bus**: [Python](azure-servicebus-py.md) | [Java](azure-servicebus-java.md) | [JS](azure-servicebus-js.md) | [.NET](azure-servicebus-dotnet.md)
## Common Issues
| Issue | Category |
|-------|----------|
| AMQP link detach, idle timeout, connection inactive | [service-troubleshooting.md](service-troubleshooting.md) |
| Message lock lost/expired, lock renewal failures | Language-specific SDK guide |
| Session lock errors, session receiver detach | Language-specific SDK guide |
| Duplicate events, checkpoint/offset reset | Language-specific SDK guide |
| Batch >1 MB rejected, partition key conflicts | [service-troubleshooting.md](service-troubleshooting.md) |
## MCP Tools
| Tool | Use |
|------|-----|
| `mcp_azure_mcp_eventhubs` | List namespaces, hubs, consumer groups |
| `mcp_azure_mcp_servicebus` | List namespaces, queues, topics, subscriptions |
| `mcp_azure_mcp_monitor` | Query diagnostic logs with KQL |
| `mcp_azure_mcp_resourcehealth` | Check service health status |
| `mcp_azure_mcp_documentation` | Search Microsoft Learn for troubleshooting docs |
auth-best-practices.md 6.0 KB
# Azure Authentication Best Practices
> Source: [Microsoft — Passwordless connections for Azure services](https://learn.microsoft.com/azure/developer/intro/passwordless-overview) and [Azure Identity client libraries](https://learn.microsoft.com/dotnet/azure/sdk/authentication/).
## Golden Rule
Use **managed identities** and **Azure RBAC** in production. Reserve `DefaultAzureCredential` for **local development only**.
## Authentication by Environment
| Environment | Recommended Credential | Why |
|---|---|---|
| **Production (Azure-hosted)** | `ManagedIdentityCredential` (system- or user-assigned) | No secrets to manage; auto-rotated by Azure |
| **Production (on-premises)** | `ClientCertificateCredential` or `WorkloadIdentityCredential` | Deterministic; no fallback chain overhead |
| **CI/CD pipelines** | `AzurePipelinesCredential` / `WorkloadIdentityCredential` | Scoped to pipeline identity |
| **Local development** | `DefaultAzureCredential` | Chains CLI, PowerShell, and VS Code credentials for convenience |
## Why Not `DefaultAzureCredential` in Production?
1. **Unpredictable fallback chain** — walks through multiple credential types, adding latency and making failures harder to diagnose.
2. **Broad surface area** — checks environment variables, CLI tokens, and other sources that should not exist in production.
3. **Non-deterministic** — which credential actually authenticates depends on the environment, making behavior inconsistent across deployments.
4. **Performance** — each failed credential attempt adds network round-trips before falling back to the next.
## Production Patterns
### .NET
```csharp
using Azure.Identity;
var credential = Environment.GetEnvironmentVariable("AZURE_FUNCTIONS_ENVIRONMENT") == "Development"
? new DefaultAzureCredential() // local dev — uses CLI/VS credentials
: new ManagedIdentityCredential(); // production — deterministic, no fallback chain
// For user-assigned identity: new ManagedIdentityCredential("<client-id>")
```
### TypeScript / JavaScript
```typescript
import { DefaultAzureCredential, ManagedIdentityCredential } from "@azure/identity";
const credential = process.env.NODE_ENV === "development"
? new DefaultAzureCredential() // local dev — uses CLI/VS credentials
: new ManagedIdentityCredential(); // production — deterministic, no fallback chain
// For user-assigned identity: new ManagedIdentityCredential("<client-id>")
```
### Python
```python
import os
from azure.identity import DefaultAzureCredential, ManagedIdentityCredential
credential = (
DefaultAzureCredential() # local dev — uses CLI/VS credentials
if os.getenv("AZURE_FUNCTIONS_ENVIRONMENT") == "Development"
else ManagedIdentityCredential() # production — deterministic, no fallback chain
)
# For user-assigned identity: ManagedIdentityCredential(client_id="<client-id>")
```
### Java
```java
import com.azure.identity.DefaultAzureCredentialBuilder;
import com.azure.identity.ManagedIdentityCredentialBuilder;
var credential = "Development".equals(System.getenv("AZURE_FUNCTIONS_ENVIRONMENT"))
? new DefaultAzureCredentialBuilder().build() // local dev — uses CLI/VS credentials
: new ManagedIdentityCredentialBuilder().build(); // production — deterministic, no fallback chain
// For user-assigned identity: new ManagedIdentityCredentialBuilder().clientId("<client-id>").build()
```
## Local Development Setup
`DefaultAzureCredential` is ideal for local dev because it automatically picks up credentials from developer tools:
1. **Azure CLI** — `az login`
2. **Azure Developer CLI** — `azd auth login`
3. **Azure PowerShell** — `Connect-AzAccount`
4. **Visual Studio / VS Code** — sign in via Azure extension
```typescript
import { DefaultAzureCredential } from "@azure/identity";
// Local development only — uses CLI/PowerShell/VS Code credentials
const credential = new DefaultAzureCredential();
```
## Environment-Aware Pattern
Detect the runtime environment and select the appropriate credential. The key principle: use `DefaultAzureCredential` only when running locally, and a specific credential in production.
> **Tip:** Azure Functions sets `AZURE_FUNCTIONS_ENVIRONMENT` to `"Development"` when running locally. For App Service or containers, use any environment variable you control (e.g. `NODE_ENV`, `ASPNETCORE_ENVIRONMENT`).
```typescript
import { DefaultAzureCredential, ManagedIdentityCredential } from "@azure/identity";
function getCredential() {
if (process.env.NODE_ENV === "development") {
return new DefaultAzureCredential(); // picks up az login / VS Code creds
}
return process.env.AZURE_CLIENT_ID
? new ManagedIdentityCredential(process.env.AZURE_CLIENT_ID) // user-assigned
: new ManagedIdentityCredential(); // system-assigned
}
```
## Security Checklist
- [ ] Use managed identity for all Azure-hosted apps
- [ ] Never hardcode credentials, connection strings, or keys
- [ ] Apply least-privilege RBAC roles at the narrowest scope
- [ ] Use `ManagedIdentityCredential` (not `DefaultAzureCredential`) in production
- [ ] Store any required secrets in Azure Key Vault
- [ ] Rotate secrets and certificates on a schedule
- [ ] Enable Microsoft Defender for Cloud on production resources
## Further Reading
- [Passwordless connections overview](https://learn.microsoft.com/azure/developer/intro/passwordless-overview)
- [Managed identities overview](https://learn.microsoft.com/entra/identity/managed-identities-azure-resources/overview)
- [Azure RBAC overview](https://learn.microsoft.com/azure/role-based-access-control/overview)
- [.NET authentication guide](https://learn.microsoft.com/dotnet/azure/sdk/authentication/)
- [Python identity library](https://learn.microsoft.com/python/api/overview/azure/identity-readme)
- [JavaScript identity library](https://learn.microsoft.com/javascript/api/overview/azure/identity-readme)
- [Java identity library](https://learn.microsoft.com/java/api/overview/azure/identity-readme)
azure-eventhubs-dotnet.md 3.5 KB
# Azure Event Hubs SDK — .NET (C#)
Package: `Azure.Messaging.EventHubs` | [README](https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/eventhub/Azure.Messaging.EventHubs/) | [Full Troubleshooting Guide](https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/eventhub/Azure.Messaging.EventHubs/TROUBLESHOOTING.md)
## Common Errors
| Exception | Reason | Fix |
|-----------|--------|-----|
| `EventHubsException` (ServiceTimeout) | Service didn't respond in time | Transient — retried automatically. Verify state if persists |
| `EventHubsException` (QuotaExceeded) | Too many active readers per consumer group | Reduce concurrent receivers or upgrade tier |
| `EventHubsException` (ConsumerDisconnected) | Higher priority consumer took ownership | Expected during load balancing; check if scaling |
| `EventHubsException` (MessageSizeExceeded) | Event too large | Reduce event payload; unlikely in practice since the client caps at the service link limit |
| `UnauthorizedAccessException` | Bad credentials | Verify connection string, SAS token, or RBAC roles |
## Exception Filtering
```csharp
try { /* receive events */ }
catch (EventHubsException ex) when (ex.Reason == EventHubsException.FailureReason.ConsumerDisconnected)
{
// Handle consumer disconnection
}
```
## Retry Configuration
Configure via `EventHubsRetryOptions` when creating the client. See [Configuring retry thresholds sample](https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/eventhub/Azure.Messaging.EventHubs/samples).
## Key Issues
- **Socket exhaustion**: Treat clients as singletons. Share `EventHubConnection` across clients if needed. Always call `CloseAsync` or `DisposeAsync`.
- **HTTP 412/409 from storage**: Normal during checkpoint store operations — not an error.
- **Partitions closing frequently**: Expected when scaling. If persists >5 min without scaling, investigate. See [Troubleshooting Guide](https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/eventhub/Azure.Messaging.EventHubs/TROUBLESHOOTING.md) for detailed diagnostics.
- **High CPU**: Limit to 1.5–3 partitions per CPU core and test at scale thoroughly if above that threshold.
- **Azure Functions**: After upgrading to v5.0+ extensions, update binding types. Reduce logging noise by filtering `Azure.Messaging.EventHubs` to Warning.
- **WebSockets**: Use `EventHubsTransportType.AmqpWebSockets` to connect over port 443 when AMQP ports (5761, 5762) are blocked.
## Checkpointing (BlobCheckpointStore)
Package: `Azure.Messaging.EventHubs.Processor` (includes `EventProcessorClient` + blob checkpoint store)
> **Auth:** `DefaultAzureCredential` is for local development. See [auth-best-practices.md](auth-best-practices.md) for production patterns.
```csharp
var credential = new DefaultAzureCredential();
var storageClient = new BlobContainerClient(
new Uri("https://<storage-account>.blob.core.windows.net/<checkpoint-container>"),
credential);
var processor = new EventProcessorClient(
storageClient,
"$Default",
"<your-namespace>.servicebus.windows.net",
"<your-eventhub>",
credential);
processor.ProcessEventAsync += async (args) =>
{
// process event
await args.UpdateCheckpointAsync();
};
```
**Common issues:**
- **Soft delete / blob versioning**: Disable both on the storage account — they cause delays during load balancing.
- **HTTP 412/409 from storage**: Normal during partition ownership negotiation; not an error.
- **Checkpoint frequency**: Call `UpdateCheckpointAsync()` per batch, not per event, to reduce storage calls.
azure-eventhubs-java.md 3.2 KB
# Azure Event Hubs SDK — Java
Package: `azure-messaging-eventhubs` | [README](https://github.com/Azure/azure-sdk-for-java/blob/main/sdk/eventhubs/azure-messaging-eventhubs/) | [Full Troubleshooting Guide](https://github.com/Azure/azure-sdk-for-java/blob/main/sdk/eventhubs/azure-messaging-eventhubs/TROUBLESHOOTING.md)
> ⚠️ **Note:** The detailed Java troubleshooting guide has moved to [Microsoft Learn](https://learn.microsoft.com/azure/developer/java/sdk/troubleshooting-messaging-event-hubs-overview).
## Common Errors
| Exception | Cause | Fix |
|-----------|-------|-----|
| `AmqpException` (connection:forced) | Idle connection disconnected | Auto-recovers; no action needed |
| `AmqpException` (unauthorized-access) | Bad credentials or missing permissions | Verify connection string, SAS, or RBAC roles |
| `AmqpException` (resource-limit-exceeded) | Too many concurrent receivers | Reduce receiver count or upgrade tier |
| `OperationTimeoutException` | Network issue or throttling | Check firewall, try AMQP over WebSockets (port 443) |
## Enable Logging
Configure via SLF4J. Add `logback-classic` dependency and set level for `com.azure.messaging.eventhubs`:
```xml
<logger name="com.azure.messaging.eventhubs" level="DEBUG"/>
```
For AMQP frame tracing:
```xml
<logger name="com.azure.core.amqp" level="DEBUG"/>
```
See [Java SDK logging docs](https://learn.microsoft.com/azure/developer/java/sdk/troubleshooting-messaging-event-hubs-overview) for details.
## Key Issues
- **High CPU / partition imbalance**: Limit to 1.5–3 partitions per CPU core.
- **Consumer disconnected**: Higher priority consumer took ownership. Expected during load balancing. Persistent issues without scaling indicate a problem.
- **Connection sharing**: Reuse `EventHubClientBuilder` connections; avoid creating new clients per operation.
## Checkpointing (BlobCheckpointStore)
Package: `azure-messaging-eventhubs-checkpointstore-blob`
> **Auth:** `DefaultAzureCredential` is for local development. See [auth-best-practices.md](auth-best-practices.md) for production patterns.
```java
TokenCredential credential = new DefaultAzureCredentialBuilder().build();
BlobContainerAsyncClient blobClient = new BlobContainerClientBuilder()
.endpoint("https://<storage-account>.blob.core.windows.net/<checkpoint-container>")
.credential(credential)
.buildAsyncClient();
EventProcessorClient processor = new EventProcessorClientBuilder()
.credential("<your-namespace>.servicebus.windows.net", "<your-eventhub>", credential)
.consumerGroup("$Default")
.checkpointStore(new BlobCheckpointStore(blobClient))
.processEvent(eventContext -> {
// process event
eventContext.updateCheckpoint();
})
.buildEventProcessorClient();
```
**Common issues:**
- **Soft delete / blob versioning**: Disable both on the storage account — they cause delays during load balancing.
- **HTTP 412/409 from storage**: Normal during partition ownership negotiation; not an error.
- **Checkpoint frequency**: Call `updateCheckpoint()` per batch, not per event, to reduce storage calls.
## Filing Issues
Include: partition count, machine specs, instance count, max heap (`-Xmx`), average `EventData` size, traffic pattern, and DEBUG-level logs (±10 min from issue).
azure-eventhubs-js.md 2.7 KB
# Azure Event Hubs SDK — JavaScript
Package: `@azure/event-hubs` | [README](https://github.com/Azure/azure-sdk-for-js/blob/main/sdk/eventhub/event-hubs/) | [Full Troubleshooting Guide](https://github.com/Azure/azure-sdk-for-js/blob/main/sdk/eventhub/event-hubs/TROUBLESHOOTING.md)
## Common Errors
| Error | Code | Fix |
|-------|------|-----|
| `MessagingError` (connection:forced) | Idle disconnect | Auto-recovers; no action needed |
| `MessagingError` (Unauthorized) | Bad credentials | Verify connection string, SAS, or RBAC roles |
| `MessagingError` (retryable: true) | Transient issue | Auto-retried per `RetryOptions`. If surfaced, all retries exhausted |
`MessagingError` fields: `name`, `code`, `retryable`, `info`, `address`, `errno`, `port`, `syscall`.
## Enable Logging
```bash
# All SDK logs
export AZURE_LOG_LEVEL=verbose
# Or use DEBUG for granular control
export DEBUG=azure*,rhea*
# Errors only
export DEBUG=azure:*:(error|warning),rhea-promise:error,rhea:events,rhea:frames,rhea:io,rhea:flow
```
Browser:
```javascript
localStorage.debug = "azure:*:info";
```
## Key Issues
- **Socket exhaustion**: Treat clients as singletons. Each new client creates a new AMQP connection/socket. Always call `close()`.
- **412 precondition failures**: Normal during subscription partition ownership negotiation.
- **Partition ownership churn**: Expected when scaling instances. Should stabilize within minutes.
- **High CPU**: Limit to 1.5–3 partitions per CPU core.
- **Subscription stops receiving**: Often a symptom of an underlying race condition during error recovery. File a GitHub issue with DEBUG logs.
- **WebSockets**: Pass `webSocketOptions` to client constructor to connect over port 443.
## Checkpointing (BlobCheckpointStore)
Package: `@azure/eventhubs-checkpointstore-blob`
> **Auth:** `DefaultAzureCredential` is for local development. See [auth-best-practices.md](auth-best-practices.md) for production patterns.
```javascript
const { BlobCheckpointStore } = require("@azure/eventhubs-checkpointstore-blob");
const { BlobServiceClient } = require("@azure/storage-blob");
const containerClient = new BlobServiceClient(storageEndpoint, credential)
.getContainerClient("checkpointstore");
const checkpointStore = new BlobCheckpointStore(containerClient);
const consumerClient = new EventHubConsumerClient(
consumerGroup, fullyQualifiedNamespace, eventHubName, credential, checkpointStore
);
```
**Common issues:**
- **Soft delete / blob versioning**: Disable both on the storage account — they cause delays during load balancing.
- **412 precondition failures**: Normal during partition ownership negotiation; not an error.
- **Checkpoint frequency**: Call `updateCheckpoint()` per batch, not per event, to reduce storage calls.
azure-eventhubs-py.md 4.3 KB
# Azure Event Hubs SDK — Python
Package: `azure-eventhub` | [README](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/eventhub/azure-eventhub) | [Full Troubleshooting Guide](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/eventhub/azure-eventhub/TROUBLESHOOTING.md)
## Common Errors
| Exception | Cause | Fix |
|-----------|-------|-----|
| `EventHubError` | Base exception wrapping AMQP errors | Check `message`, `error`, `details` fields |
| `ConnectionLostError` | Idle connection disconnected | Auto-recovers on next operation; no action needed |
| `AuthenticationError` | Bad credentials or expired SAS | Regenerate key, check RBAC roles, verify connection string |
| `OperationTimeoutError` | Network or throttling | Check firewall, try WebSockets (port 443), increase timeout |
## Retry Configuration
> **Auth:** `DefaultAzureCredential` is for local development. See [auth-best-practices.md](auth-best-practices.md) for production patterns.
```python
from azure.eventhub import EventHubProducerClient
from azure.identity import DefaultAzureCredential
client = EventHubProducerClient(
fully_qualified_namespace="<your-namespace>.servicebus.windows.net",
eventhub_name="<your-eventhub>",
credential=DefaultAzureCredential(),
retry_total=3,
retry_backoff_factor=0.8,
retry_backoff_max=120,
retry_mode='exponential'
)
```
## Consumer Client Retry Configuration
> **Auth:** `DefaultAzureCredential` is for local development. See [auth-best-practices.md](auth-best-practices.md) for production patterns.
Under heavy load, tune the retry policy on `EventHubConsumerClient` to reduce timeouts:
| Parameter | Default | Description |
|-----------|---------|-------------|
| `retry_total` | 3 | Max retry attempts per operation |
| `retry_backoff_factor` | 0.8 | Backoff multiplier between retries (seconds) |
| `retry_backoff_max` | 120 | Max backoff interval (seconds) |
| `retry_mode` | `exponential` | `fixed` or `exponential` |
```python
from azure.eventhub import EventHubConsumerClient
from azure.eventhub.extensions.checkpointstoreblob import BlobCheckpointStore
from azure.identity import DefaultAzureCredential
credential = DefaultAzureCredential()
checkpoint_store = BlobCheckpointStore(
blob_account_url="https://<storage-account>.blob.core.windows.net",
container_name="<checkpoint-container>",
credential=credential
)
client = EventHubConsumerClient(
fully_qualified_namespace="<your-namespace>.servicebus.windows.net",
eventhub_name="<your-eventhub>",
consumer_group="$Default",
credential=credential,
checkpoint_store=checkpoint_store,
retry_total=5,
retry_backoff_factor=1.0,
retry_backoff_max=120,
retry_mode='exponential'
)
```
## Enable Logging
```python
import logging, sys
handler = logging.StreamHandler(stream=sys.stdout)
handler.setFormatter(logging.Formatter("%(asctime)s | %(threadName)s | %(levelname)s | %(name)s | %(message)s"))
logger = logging.getLogger('azure.eventhub')
logger.setLevel(logging.DEBUG)
logger.addHandler(handler)
# Enable AMQP frame tracing
client = EventHubProducerClient(..., logging_enable=True)
```
## Key Issues
- **Buffered producer not sending**: Ensure enough `ThreadPoolExecutor` workers (one per partition). Use `buffer_concurrency` kwarg.
- **Blocking calls in async**: Run CPU-bound code in an executor; blocking the event loop impacts load balancing and checkpointing.
- **Consumer disconnected**: Expected during load balancing. If persistent with no scaling, file an issue.
- **Soft delete on checkpoint store**: Disable "soft delete" and "blob versioning" on the storage account used for checkpointing.
- **Always close clients**: Use `with` statement or call `close()` to avoid socket/connection leaks.
## Checkpointing (BlobCheckpointStore)
Package: `azure-eventhub-checkpointstoreblob` (sync) / `azure-eventhub-checkpointstoreblob-aio` (async)
See the [Consumer Client Retry Configuration](#consumer-client-retry-configuration) section above for a full `EventHubConsumerClient` example with `BlobCheckpointStore`.
**Common issues:**
- **Soft delete / blob versioning**: Disable both on the storage account — they cause large delays during load balancing.
- **HTTP 412/409 from storage**: Normal during partition ownership negotiation; not an error.
- **Checkpoint frequency**: Checkpoint after processing each batch, not each event, to avoid storage throttling.
azure-servicebus-dotnet.md 2.5 KB
# Azure Service Bus SDK — .NET (C#)
Package: `Azure.Messaging.ServiceBus` | [README](https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/servicebus/Azure.Messaging.ServiceBus/) | [Full Troubleshooting Guide](https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/servicebus/Azure.Messaging.ServiceBus/TROUBLESHOOTING.md)
## Common Errors
| Exception | Reason | Fix |
|-----------|--------|-----|
| `ServiceBusException` (ServiceTimeout) | Service didn't respond | Transient — auto-retried. For session accept, means no unlocked sessions |
| `ServiceBusException` (MessageLockLost) | Lock expired or link detached | Renew lock, reduce processing time, check network |
| `ServiceBusException` (SessionLockLost) | Session lock expired | Re-accept session, renew lock before expiry |
| `ServiceBusException` (QuotaExceeded) | Too many concurrent receives | Reduce receivers or use batch receives |
| `ServiceBusException` (MessageSizeExceeded) | Message or batch too large | Reduce payload. Premium tier supports individual messages up to 100MB. Batch limit is artificially computed on the client from the max message size sent by the service, so batches can also be impacted |
| `ServiceBusException` (ServiceBusy) | Request throttled | Auto-retried with 10s backoff. See [throttling docs](https://learn.microsoft.com/azure/service-bus-messaging/service-bus-throttling) |
| `UnauthorizedAccessException` | Bad credentials | Verify connection string, SAS, or RBAC roles |
## Exception Filtering
```csharp
try { /* receive messages */ }
catch (ServiceBusException ex) when (ex.Reason == ServiceBusFailureReason.ServiceTimeout)
{
// Handle timeout
}
```
## Key Issues
- **Socket exhaustion**: Treat `ServiceBusClient` as singleton. Each creates a new AMQP connection. Always call `CloseAsync`/`DisposeAsync`.
- **Lock lost before expiry**: Can happen on link detach (transient network) or 10-min idle timeout.
- **Processor high concurrency**: May cause hangs with extreme concurrency settings. Test with moderate values.
- **Session processor slow switching**: Tune `SessionIdleTimeout` to reduce wait time between sessions.
- **Batch size limits**: Batch limit is artificially computed on the client from the max message size sent by the service. Send large messages individually if batch creation fails.
- **Transactions across entities**: Requires all entities on same namespace. Use `ServiceBusClient.CreateSender` with `via` entity support.
- **WebSockets**: Use `ServiceBusTransportType.AmqpWebSockets` when AMQP ports (5761, 5762) are blocked.
azure-servicebus-java.md 2.2 KB
# Azure Service Bus SDK — Java
Package: `azure-messaging-servicebus` | [README](https://github.com/Azure/azure-sdk-for-java/blob/main/sdk/servicebus/azure-messaging-servicebus/) | [Full Troubleshooting Guide](https://github.com/Azure/azure-sdk-for-java/blob/main/sdk/servicebus/azure-messaging-servicebus/TROUBLESHOOTING.md)
## Common Errors
| Exception | Cause | Fix |
|-----------|-------|-----|
| `AmqpException` (unauthorized-access) | Bad credentials or missing permissions | Verify connection string, SAS, or RBAC roles |
| `AmqpException` (connection:forced) | Idle connection or transient network issue | Auto-recovers; no action needed |
| `ServiceBusException` (MESSAGE_LOCK_LOST) | Lock expired during processing | Reduce processing time, disable auto-complete, settle manually |
## Key Issues
### Processor hangs with high prefetch + maxConcurrentCalls
`Update disposition request timed out.` — Client stops processing new messages.
**Cause**: Thread starvation when thread pool size ≤ `maxConcurrentCalls`.
**Fix**:
```bash
# Increase reactor thread pool
-Dreactor.schedulers.defaultBoundedElasticSize=<value greater than concurrency>
```
Also set `prefetchCount(0)` to disable prefetch. This is more frequent in AKS environments.
### Implicit prefetch in ServiceBusReceiverClient
Even with prefetch disabled in the builder, `receiveMessages` API can re-enable prefetch implicitly. See [SyncReceiveAndPrefetch](https://github.com/Azure/azure-sdk-for-java/tree/main/sdk/servicebus/azure-messaging-servicebus/docs/SyncReceiveAndPrefetch.md).
### Autocomplete issues
Autocomplete and auto-lock-renewal have known issues with buffered/prefetched messages.
**Fix**: Use `disableAutoComplete()` and `.maxAutoLockRenewalDuration(Duration.ZERO)`, then settle messages explicitly.
## Enable Logging
Configure via SLF4J:
```xml
<logger name="com.azure.messaging.servicebus" level="DEBUG"/>
```
See [Java SDK logging docs](https://learn.microsoft.com/azure/developer/java/sdk/troubleshooting-messaging-service-bus-overview) for details.
## Filing Issues
Include: namespace tier, entity type/config, machine specs, max heap (`-Xmx`), `maxConcurrentCalls`, `prefetchCount`, autoComplete setting, traffic pattern, and DEBUG-level logs (±10 min from issue).
azure-servicebus-js.md 2.4 KB
# Azure Service Bus SDK — JavaScript
Package: `@azure/service-bus` | [README](https://github.com/Azure/azure-sdk-for-js/blob/main/sdk/servicebus/service-bus/) | [Full Troubleshooting Guide](https://github.com/Azure/azure-sdk-for-js/blob/main/sdk/servicebus/service-bus/TROUBLESHOOTING.md)
## Common Errors
| Error Code | Cause | Fix |
|------------|-------|-----|
| `ServiceTimeout` | Service didn't respond; or no unlocked sessions | Transient — auto-retried. Verify state if persists |
| `MessageLockLost` | Processing exceeded lock duration or link detached | Reduce processing time, ensure autolock renewal works |
| `SessionLockLost` | Session lock expired or link detached | Re-accept session, keep renewing lock |
| `QuotaExceeded` | Too many concurrent receives | Reduce receivers or use batch receives |
| `MessageSizeExceeded` | Message or batch > max size | Reduce payload. Premium supports individual messages up to 100MB. Batch limit is computed from max message size on the client, so batches can also be impacted |
| `UnauthorizedAccess` | Bad credentials | Verify connection string, SAS, or RBAC roles |
`ServiceBusError` fields: `code`, `retryable`, `name`, `info`, `address`.
## Enable Logging
```bash
# All SDK logs
export AZURE_LOG_LEVEL=verbose
# Or granular control
export DEBUG=azure*,rhea*
# Errors only
export DEBUG=azure:service-bus:error,azure:core-amqp:error,rhea-promise:error,rhea:events,rhea:frames,rhea:io,rhea:flow
```
Log to file:
```bash
node app.js > out.log 2>debug.log
```
## Key Issues
- **Socket exhaustion**: Treat `ServiceBusClient` as singleton. Each creates a new AMQP connection. Always call `close()`.
- **Lock lost before expiry**: Can happen on link detach (transient network issue or 10-min idle timeout). Not always due to processing time.
- **Batch receive returns fewer messages**: After first message arrives, receiver waits only 1s for additional messages. `maxWaitTimeInMs` controls wait for the *first* message only.
- **Autolock renewal not working**: Ensure system clock is accurate. Autolock relies on system time.
- **Batch size limits**: Batch limit is artificially computed on the client from the max message size sent by the service. Send large messages individually if batch creation fails.
- **WebSockets**: Pass `webSocketOptions` to `ServiceBusClient` constructor for port 443 connectivity.
- **Distributed tracing**: Experimental OpenTelemetry support via `@azure/opentelemetry-instrumentation-azure-sdk`.
azure-servicebus-py.md 2.3 KB
# Azure Service Bus SDK — Python
Package: `azure-servicebus` | [README](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/servicebus/azure-servicebus/) | [Full Troubleshooting Guide](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/servicebus/azure-servicebus/TROUBLESHOOTING.md)
## Common Errors
| Exception | Cause | Fix |
|-----------|-------|-----|
| `ServiceBusAuthenticationError` | Invalid credentials | Check connection string, regenerate SAS key |
| `ServiceBusAuthorizationError` | Missing Send/Listen claim | Assign `Azure Service Bus Data Owner/Sender/Receiver` RBAC role |
| `ServiceBusConnectionError` | Network or firewall | Check AMQP port 5671, try `TransportType.AmqpOverWebsocket` |
| `OperationTimeoutError` | Service didn't respond in time | Adjust retry config, verify network |
| `MessageLockLostError` | Processing exceeded lock duration | Use `AutoLockRenewer`, reduce processing time |
| `SessionLockLostError` | Session lock expired | Reconnect to session, keep renewing lock |
| `MessageSizeExceededError` | Message or batch too large | Reduce payload. Premium supports individual messages up to 100MB. Batch limit is computed from max message size on the client, so batches can also be impacted |
## Enable Logging
```python
import logging, sys
handler = logging.StreamHandler(stream=sys.stdout)
handler.setFormatter(logging.Formatter("%(asctime)s | %(threadName)s | %(levelname)s | %(name)s | %(message)s"))
logger = logging.getLogger('azure.servicebus')
logger.setLevel(logging.DEBUG)
logger.addHandler(handler)
# Enable AMQP frame tracing
from azure.servicebus import ServiceBusClient
client = ServiceBusClient(..., logging_enable=True)
```
## AutoLockRenewer
```python
from azure.servicebus import AutoLockRenewer
renewer = AutoLockRenewer()
with receiver:
for message in receiver.receive_messages(max_message_count=10):
renewer.register(receiver, message, max_lock_renewal_duration=300)
# process message
receiver.complete_message(message)
```
## Key Issues
- **Mixing sync/async**: Don't use `time.sleep()` in async code; use `await asyncio.sleep()`.
- **Dead letter debugging**: Use `sub_queue=ServiceBusSubQueue.DEAD_LETTER` to inspect `dead_letter_reason` and `dead_letter_error_description`.
- **Always close clients**: Use `with` statement or call `close()` to avoid connection leaks.
service-troubleshooting.md 5.0 KB
# Service-Level Troubleshooting
Covers connectivity, firewall, and network issues that apply regardless of SDK language.
## Permanent Connectivity Issues
If the client **cannot connect at all**:
1. **Verify connection string** — Get from Azure portal. For **Event Hubs (Kafka endpoint)** clients, also check `producer.config` / `consumer.config`.
2. **Check service outage** — [Azure status page](https://azure.status.microsoft/status)
3. **Firewall / ports** — Open AMQP 5671 and 5672, HTTPS 443. For **Event Hubs (Kafka endpoint)** only, also open Kafka 9093. Use WebSockets (port 443) as fallback.
4. **IP firewall** — If enabled on namespace, ensure client IP is allowed.
5. **VNet / private endpoints** — Confirm app runs in correct subnet. Check service endpoint and NSG rules.
6. **Proxy / SSL** — Intercepting proxies can cause SSL handshake failures. Test with proxy disabled.
### Quick Connectivity Test
Run the connectivity probe script. It resolves DNS, tests HTTPS reachability, and probes the messaging ports (AMQP `5671`/`5672`, HTTPS `443`), returning a normalized report instead of raw `curl`/`nslookup` output. Add `--kafka` / `-Kafka` to also probe the Event Hubs Kafka port `9093`.
Scripts (paths below are relative to the skill root, `plugins/azure-skills/skills/azure-diagnostics`, so run them from there): [`scripts/test-messaging-connectivity.sh`](../../scripts/test-messaging-connectivity.sh) (bash) and [`scripts/test-messaging-connectivity.ps1`](../../scripts/test-messaging-connectivity.ps1) (PowerShell).
```powershell
# from plugins/azure-skills/skills/azure-diagnostics
.\scripts\test-messaging-connectivity.ps1 -Namespace <namespace>
```
```bash
# from plugins/azure-skills/skills/azure-diagnostics
bash ./scripts/test-messaging-connectivity.sh <namespace>
```
The namespace may be a full FQDN or a bare name (`.servicebus.windows.net` is appended automatically).
**Example (Event Hubs, including Kafka):**
```bash
# from plugins/azure-skills/skills/azure-diagnostics
bash ./scripts/test-messaging-connectivity.sh contoso.servicebus.windows.net --kafka
```
## Transient Connectivity Issues
If connectivity is **intermittent**:
1. **Upgrade SDK** — Use latest version; transient issues may already be fixed.
2. **Check dropped packets** — `netstat -s` (Linux) or `netsh interface ipv4 show subinterface` (Windows).
3. **Capture network traces** — Use Wireshark or `tcpdump` filtered on namespace IP.
4. **Idle disconnect** — Service disconnects idle AMQP connections. Clients auto-reconnect; this is expected.
## WebSocket Configuration by Language
| Language | Setting |
|----------|---------|
| .NET | `EventHubsTransportType.AmqpWebSockets` / `ServiceBusTransportType.AmqpWebSockets` |
| Java | `AmqpTransportType.AMQP_WEB_SOCKETS` |
| Python | `transport_type=TransportType.AmqpOverWebsocket` |
| JavaScript | `webSocketOptions` in client constructor |
## Authentication Checklist
| Issue | Fix |
|-------|-----|
| Invalid connection string | Re-copy from Azure portal |
| Expired SAS token | Regenerate or increase validity |
| Missing RBAC role | Assign the corresponding *Azure Event Hubs Data Owner/Sender/Receiver* or *Azure Service Bus Data Owner/Sender/Receiver* role |
| Managed Identity not configured | Enable system/user-assigned identity, assign role on namespace |
## Sender Issues (All Languages)
- **Batch >1MB fails** — Service rejects batches over 1MB even with Premium large message support. Send large messages individually.
- **Multiple partition keys in batch** — Not allowed. Group messages by `partitionKey` (or `sessionId`) into separate batches.
## Receiver Issues (All Languages)
- **Batch receive returns fewer messages** — After the first message arrives, the receiver waits briefly (20ms–1s depending on SDK) for more. `maxWaitTime` only controls the wait for the *first* message.
- **Lock lost before expiry** — Can occur on AMQP link detach (transient network or 10-min idle timeout), not only when processing exceeds lock duration.
- **Socket exhaustion** — Treat clients as singletons. Each new client creates a new AMQP connection. Always close/dispose clients when done.
## Further Reading
- [Event Hubs troubleshooting guide](https://learn.microsoft.com/azure/event-hubs/troubleshooting-guide)
- [Service Bus troubleshooting guide](https://learn.microsoft.com/azure/service-bus-messaging/service-bus-troubleshooting-guide)
- [Event Hubs quotas and limits](https://learn.microsoft.com/azure/event-hubs/event-hubs-quotas)
- [Service Bus quotas and limits](https://learn.microsoft.com/azure/service-bus-messaging/service-bus-quotas)
- [Event Hubs AMQP troubleshooting](https://learn.microsoft.com/azure/event-hubs/event-hubs-amqp-troubleshoot)
- [Service Bus AMQP troubleshooting](https://learn.microsoft.com/azure/service-bus-messaging/service-bus-amqp-troubleshoot)
- [Event Hubs IP addresses and service tags](https://learn.microsoft.com/azure/event-hubs/troubleshooting-guide#what-ip-addresses-do-i-need-to-allow)
- [Service Bus IP addresses](https://learn.microsoft.com/azure/service-bus-messaging/service-bus-faq#what-ip-addresses-do-i-need-to-add-to-allowlist-)
License (MIT)
View full license text
MIT License Copyright 2025 (c) Microsoft Corporation. Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.