Chapter 9

Monitoring: metrics, logs, alerts and Network Watcher

You've built a small but real environment. Now you need to see what it does and hear when something goes wrong. This chapter covers Azure Monitor (metrics, logs, KQL, alerts, Insights) and Network Watcher. Everything in the lab points at resources you already have.

Azure Monitor in one picture

Azure Monitor is the umbrella for collecting, analysing and acting on telemetry. It works with two main kinds of data, and almost every question in this area comes down to "is this a metric or a log, and how does it get there?"

Metrics

Platform metrics (like a VM's Percentage CPU or a storage account's Transactions) are collected automatically, every minute, and kept for 93 days. You don't configure anything. Note that a VM's platform metrics come from the host; metrics from inside the OS (memory used, disk free space) need the Azure Monitor Agent.

In Metrics explorer you build a chart by picking a scope, a metric and an aggregation. Reading those charts correctly is a skill the exam tests:

• Aggregation: Avg, Min, Max, Sum, Count. A CPU chart with Avg can hide short spikes that Max shows.
• Time granularity: each point summarises 1 minute, 5 minutes, 1 hour… Coarse granularity smooths the line.
• Dimensions: many metrics can be split or filtered by a dimension, for example load balancer Health Probe Status per Backend IP, or storage Transactions per API name or Response type.
• Charts can be pinned to a dashboard, added to a workbook, or turned into an alert rule.

Logs and Log Analytics

Types of logs

• Activity log: who did what on the control plane of a subscription (create VM, delete NSG, assign role, policy denials, service health). Collected automatically and kept for 90 days. Send it to a workspace with a diagnostic setting to keep it longer and query it with KQL (table AzureActivity).
• Resource logs: what happens inside a resource (blob reads, Key Vault access, load balancer events). Not collected until you create a diagnostic setting.
• Microsoft Entra logs: sign-in and audit logs of the tenant.
• Guest OS logs: Windows event logs, Linux syslog, performance counters, through the Azure Monitor Agent (AMA) and a data collection rule (DCR) that says what to collect and where to send it.

Diagnostic settings

A diagnostic setting on a resource (or subscription) sends selected log categories and/or metrics to one or more destinations:

• Log Analytics workspace: to query with KQL, alert on, and use in Insights.
• Storage account: cheap long-term archive (e.g. for auditors).
• Event Hubs: to stream to a SIEM or a third-party tool.
• Partner solutions (e.g. Datadog, Elastic).

A resource can have up to five diagnostic settings. For storage accounts, diagnostic settings are per service (blob, file, queue, table). At scale, you deploy diagnostic settings with Azure Policy (DeployIfNotExists, chapter 2).

Log Analytics workspaces

A Log Analytics workspace stores log data in tables (AzureActivity, Heartbeat, Perf, StorageBlobLogs…). Things to know:

• Retention: interactive (analytics) retention from 30 days to 2 years, plus long-term retention up to 12 years at a lower price. The first 31 days are included in the ingestion price.
• Pricing: you pay mainly per GB ingested. Commitment tiers give a discount; a daily cap stops runaway costs (and data).
• Table plans: Analytics (full features), Basic and Auxiliary (cheaper, limited queries) for high-volume, low-value logs.
• Access control: workspace-context (you see all data you have workspace rights to) or resource-context (you see only logs of resources you have access to). Use few workspaces; many organisations use one per region or per environment.

Querying with KQL

Kusto Query Language (KQL) reads like a pipeline: start with a table, then pass the rows through operators with |. You don't need to be an expert, but you must be able to read a query and say what it returns.

// How many operations did each person do in the last day?
AzureActivity
| where TimeGenerated > ago(1d)
| where ActivityStatusValue == "Success"
| summarize Operations = count() by Caller
| order by Operations desc
| take 10

String comparisons: == is case-sensitive, =~ isn't; contains matches substrings, has matches whole terms (and is faster).

Alerts

An alert rule has four parts:

1. Scope: the resource(s) to watch. A metric alert can cover many VMs of one type in one region at once.
2. Condition: the signal and its logic. Signal types:
   • Metric: "avg CPU > 80% over 5 minutes". Static threshold or dynamic thresholds (machine learning learns what's normal).
   • Log search: a KQL query, run on a schedule, that fires when results meet a condition.
   • Activity log: "someone deleted a VM", "a policy denied a request", Service Health (Azure incidents and planned maintenance in your regions) and Resource Health (your specific resource became unavailable).
3. Actions: one or more action groups.
4. Details: name and severity from Sev 0 (critical) to Sev 4 (verbose).

Metric alerts are stateful by default: they fire once, and resolve automatically when the condition is no longer true. An alert instance has a monitor condition (Fired / Resolved) and a user response you set (New, Acknowledged, Closed).

Action groups

An action group is a reusable list of who to notify and what to run:

• Notifications: email, SMS, Azure mobile app push, voice call, and Email Azure Resource Manager role (e.g. all Owners of the subscription).
• Actions: Automation runbook, Azure Function, Logic App, webhook / secure webhook, ITSM connector, Event Hub.

There are rate limits: for example no more than one SMS every 5 minutes and about 100 emails per hour per address. So an alert storm doesn't flood your phone.

Alert processing rules

Alert processing rules act on alerts after they fire, without editing each alert rule. Two uses:

• Suppression: remove all action groups during a planned maintenance window ("no notifications for rg-az104-lab on Saturday 22:00–02:00"). The alerts are still created, you just aren't notified.
• Add action groups at scale: "send every Sev 0 alert in this subscription to the on-call team as well".

A rule has a scope (subscription, resource group, resources), optional filters (severity, alert rule name, resource type, monitor service…) and an optional schedule (once, daily, weekly).

Insights

Insights are ready-made monitoring experiences for specific resource types, built on metrics, logs and workbooks:

• VM insights: CPU, memory, disk and network performance per VM, and (optionally) a Map of processes and connections. Enabling it installs the Azure Monitor Agent and creates a data collection rule that sends data to a workspace.
• Storage insights: capacity, transactions, latency and availability across all your storage accounts. Works from metrics, nothing to enable.
• Network insights: topology, health of load balancers, gateways, and connectivity.
• Container insights for AKS and Container Apps; Application Insights for code-level performance and failures of web apps.

Network Watcher

Network Watcher is a regional service for diagnosing IaaS network problems. It's enabled automatically per region (in a resource group called NetworkWatcherRG). Know which tool answers which question:

Flow logs record IP traffic. VNet flow logs are the current version; the older NSG flow logs can no longer be created new and are being retired. Flow logs are written to a storage account, and Traffic analytics processes them into a Log Analytics workspace for dashboards.

Connection Monitor runs continuous tests from sources (Azure VMs, scale sets, on-premises machines with the agent) to destinations (VMs, URLs, IPs, FQDNs) over TCP, HTTP or ICMP. It reports checks failed % and round-trip time, and you can alert on them.

Self-check

Everything here uses log-az104 (chapter 4), the VMs (chapter 5), the load balancer (chapter 6), the storage account (chapter 7) and the route from chapter 3. Start the VMs and paste the variables:

RG=rg-az104-lab; LOC=westeurope; LOC2=northeurope; INIT=xx
ST=staz104${INIT}01
az vm start -g $RG -n vm-web1 --no-wait; az vm start -g $RG -n vm-web2 --no-wait
LAWID=$(az monitor log-analytics workspace show -g $RG -n log-az104 --query id -o tsv)

1. Task

   Read the metrics

   Build one chart with the CPU of both web VMs as separate lines over the last 4 hours, and decide whether Avg or Max aggregation is more useful to spot short spikes. Build a second chart for lbe-web showing Data Path Availability. Pin both to a dashboard.

   Done when

   • A dashboard shows both charts.
   Solution · Portal

   1. Monitor → Metrics → scope: select both VMs (same type and region) → metric Percentage CPU, aggregation Max → Apply splitting by Resource (if offered) or add a second metric line.
   2. Time range Last 4 hours. Save to dashboard → Pin to dashboard (new dashboard az104).
   3. New chart: scope lbe-web, metric Data Path Availability, aggregation Avg → pin.

   Max shows short spikes; Avg over 5 minutes smooths them away.

2. Task

   Collect the logs that aren't collected by default

   Send to log-az104:

   • the subscription's activity log (all categories);
   • the blob service logs of your storage account (read, write, delete) plus its transaction metrics.

   Then read a blob (e.g. download hello.txt in the portal's storage browser) so there's something to see.

   Done when

   • Both diagnostic settings exist.
   • After 5–15 minutes, the tables AzureActivity and StorageBlobLogs have rows.
   Solution · Portal

   1. Monitor → Activity log → Export Activity Logs → Add diagnostic setting: tick all categories, destination Send to Log Analytics workspace → log-az104.
   2. Storage account → Monitoring → Diagnostic settings → blob → Add diagnostic setting: StorageRead, StorageWrite, StorageDelete, metric Transaction → log-az104.
   Solution · Azure CLI

   az monitor diagnostic-settings subscription create -n diag-activity --location $LOC --workspace $LAWID --logs '[
    {"category":"Administrative","enabled":true},{"category":"Security","enabled":true},
    {"category":"ServiceHealth","enabled":true},{"category":"Alert","enabled":true},
    {"category":"Recommendation","enabled":true},{"category":"Policy","enabled":true},
    {"category":"Autoscale","enabled":true},{"category":"ResourceHealth","enabled":true}]'
   
   STID=$(az storage account show -g $RG -n $ST --query id -o tsv)
   az monitor diagnostic-settings create -n diag-blob --resource $STID/blobServices/default --workspace $LAWID \
     --logs '[{"category":"StorageRead","enabled":true},{"category":"StorageWrite","enabled":true},{"category":"StorageDelete","enabled":true}]' \
     --metrics '[{"category":"Transaction","enabled":true}]'

3. Task

   Look inside the VMs

   Enable VM insights for both web VMs, sending data to log-az104. Find out which extension got installed and which data collection rule was created.

   Done when

   • Monitor → Virtual Machines → Performance shows CPU and memory for both VMs (allow 10–15 minutes).
   • You can name the agent extension and the DCR.
   Solution · Portal

   1. vm-web1 → Monitoring → Insights → Enable → choose Azure Monitor Agent, create or pick a data collection rule (e.g. MSVMI-az104) targeting log-az104; process/dependency map optional → Configure.
   2. Repeat for vm-web2 with the same DCR.
   3. The VM's Extensions + applications shows AzureMonitorLinuxAgent; the DCR is under Monitor → Data Collection Rules.
4. Task

   Get an email when a web server is overloaded

   Create an action group ag-az104-email that emails you. Then create one metric alert rule alert-vm-cpu that covers both web VMs and fires at Sev 2 when average CPU is above 80% over 5 minutes. Trigger it by loading the CPU of vm-web1.

   Done when

   • You receive the "Fired" email, and later a "Resolved" email.
   • Monitor → Alerts shows the alert; you change its state to Acknowledged.
   Hint

   For load: Run command timeout 900 sh -c 'yes > /dev/null' keeps one core at 100% for 15 minutes.

   Solution · Portal

   1. Monitor → Alerts → Action groups → Create: name ag-az104-email, display name az104, Notifications: Email/SMS/Push/Voice → your email.
   2. Monitor → Alerts → Create → Alert rule: scope → filter resource type Virtual machines, select both VMs. Condition Percentage CPU, static, Average, greater than 80, check every 1 min, lookback 5 min. Actions: ag-az104-email. Details: Sev 2, name alert-vm-cpu.
   3. vm-web1 → Run command → timeout 900 sh -c 'yes > /dev/null'.
   Solution · Azure CLI

   az monitor action-group create -g $RG -n ag-az104-email --short-name az104 --action email me <your-email>
   VM1=$(az vm show -g $RG -n vm-web1 --query id -o tsv); VM2=$(az vm show -g $RG -n vm-web2 --query id -o tsv)
   az monitor metrics alert create -g $RG -n alert-vm-cpu --scopes $VM1 $VM2 \
     --target-resource-type Microsoft.Compute/virtualMachines --region $LOC \
     --condition "avg Percentage CPU > 80" --window-size 5m --evaluation-frequency 1m \
     --severity 2 --action ag-az104-email
   az vm run-command invoke -g $RG -n vm-web1 --command-id RunShellScript \
     --scripts "nohup timeout 900 sh -c 'yes > /dev/null' >/dev/null 2>&1 &"

5. Task

   Silence alerts during maintenance

   Next Saturday from 22:00 to 23:00 you'll patch the VMs. Make sure no alert in rg-az104-lab sends notifications in that window, without changing or disabling any alert rule.

   Done when

   • An alert processing rule with a one-time schedule and suppression exists for the resource group.
   Solution · Portal

   Monitor → Alerts → Alert processing rules → Create: scope rg-az104-lab, rule type Suppress notifications, scheduling At a specific time, start Saturday 22:00, end 23:00 (your time zone), name apr-maintenance.

   Solution · Azure CLI

   RGID=$(az group show -n $RG --query id -o tsv)
   # change the dates to your next Saturday
   az monitor alert-processing-rule create -g $RG -n apr-maintenance --rule-type RemoveAllActionGroups \
     --scopes $RGID --schedule-start-datetime "2026-11-21 22:00:00" --schedule-end-datetime "2026-11-21 23:00:00" \
     --schedule-time-zone "W. Europe Standard Time" --description "Patch window"

6. Task

   Answer questions with KQL

   In log-az104 → Logs, write queries that answer:

   1. When did each VM last send a heartbeat?
   2. What was the CPU utilisation per VM in 5-minute buckets over the last hour, as a chart? (You should see your load test.)
   3. Who did which operations in your subscription today?
   4. Which blob operations hit your storage account, with status and caller IP?
   Done when

   • All four queries return results, and query 2 shows a time chart with the CPU spike.
   Hint

   Tables: Heartbeat, InsightsMetrics (VM insights; namespace Processor, name UtilizationPercentage), AzureActivity, StorageBlobLogs.

   Solution · KQL

   // 1
   Heartbeat
   | summarize LastSeen = max(TimeGenerated) by Computer
   
   // 2
   InsightsMetrics
   | where TimeGenerated > ago(1h)
   | where Namespace == "Processor" and Name == "UtilizationPercentage"
   | summarize CPU = avg(Val) by bin(TimeGenerated, 5m), Computer
   | render timechart
   
   // 3
   AzureActivity
   | where TimeGenerated > ago(1d)
   | summarize Operations = count() by Caller, OperationNameValue
   | order by Operations desc
   
   // 4
   StorageBlobLogs
   | where TimeGenerated > ago(1d)
   | project TimeGenerated, OperationName, StatusCode, CallerIpAddress, Uri
   | order by TimeGenerated desc

7. Task

   Diagnose the network with Network Watcher

   • Prove that traffic from vm-web1 to 203.0.113.10 is dropped, and name the route that drops it.
   • Check right now whether vm-web1 can reach your storage account's file endpoint on port 445, and which IP it connects to.
   • Set up continuous monitoring (every 30 seconds) from vm-web1 to the file endpoint on 445 and to www.microsoft.com on 443, named cm-az104.
   Done when

   • Next hop: type None, route blackhole-testnet (from chapter 3).
   • Connection troubleshoot: Reachable, via 10.20.2.x (the private endpoint).
   • cm-az104 shows checks and round-trip times for both destinations.
   Solution · Portal

   1. Network Watcher → Next hop: VM vm-web1, source IP its private IP, destination 203.0.113.10.
   2. Network Watcher → Connection troubleshoot: source vm-web1, destination Manually staz104xx01.file.core.windows.net, TCP port 445. (It installs the Network Watcher agent if needed.)
   3. Network Watcher → Connection monitor → Create: name cm-az104, region; test group with source vm-web1, destinations: External addresses staz104xx01.file.core.windows.net and www.microsoft.com; test configurations TCP 445 and HTTPS 443, frequency 30 s; workspace log-az104.
   Solution · Azure CLI

   IP1=$(az vm show -g $RG -n vm-web1 -d --query privateIps -o tsv)
   az network watcher show-next-hop -g $RG --vm vm-web1 --source-ip $IP1 --dest-ip 203.0.113.10
   az network watcher test-connectivity -g $RG --source-resource vm-web1 \
     --dest-address $ST.file.core.windows.net --dest-port 445 \
     --query "{status:connectionStatus, hops:hops[].address}"
   # Connection Monitor is easiest in the portal (see the portal solution).