Setting Up Azure Monitor Alerts and Action Groups

Everything I've deployed to Azure so far, I've found out about problems by opening it and looking. That works for a study project with no users. It obviously doesn't work for anything real, which is what got me into this part of AZ-900.

The two halves of an alert

The thing that clicked for me is that Azure splits alerting into two separate pieces, and they do different jobs.

An alert rule is the condition. It watches a metric or a log and decides when something counts as wrong: CPU above 80% for five minutes, HTTP 5xx responses above some count, a VM that stopped responding.

An action group is what happens next. It's the list of who gets notified and how: send an email, send an SMS, call a webhook, trigger a Logic App or a Function.

Keeping them separate seemed like unnecessary indirection until I thought about the alternative. If the notification details lived inside each alert rule, then changing who gets paged means editing every rule you've ever created. With action groups, the contact list is defined once and referenced by every rule that needs it. Change it in one place, every alert follows.

What you can actually alert on

Two broad sources, and the difference matters.

Metric alerts run on numeric data that resources emit continuously: CPU percentage, memory, request count, response time, disk usage. These evaluate quickly and are the right tool for "is this resource healthy right now."

Log alerts run a query against collected logs on a schedule. Slower, more flexible, and the right tool for things a metric can't express, like "more than ten failed login attempts from the same IP in five minutes."

For the kind of applications I build, the metric ones cover most of what I'd want. Response time going up and error rate going up are the two signals that tell me something's wrong before a user tells me.

Alerts have severity, and it's not decoration

Alert rules get a severity from 0 to 4, where 0 is critical and 4 is verbose. It looks like metadata you'd fill in and ignore.

It isn't, because it's what lets you route different urgencies to different action groups. A severity 0 alert can trigger SMS at 3am. A severity 3 can send an email that gets read on Monday. Without severities, everything is equally urgent, which in practice means nothing is, because people start ignoring the channel.

The failure mode is too many alerts, not too few

This was the most useful thing I took from studying it, and it's not a technical detail.

An alert that fires constantly gets ignored. Once a team learns that a particular alert is usually noise, they stop reading it, and then it's worse than having no alert at all, because everyone believes something is being watched when nothing effectively is.

So the useful question when creating an alert isn't "would I like to know about this." It's "if this fires, is there something I would actually do?" If the honest answer is no, the alert is noise wearing a warning label. A dashboard is the right place for information you want to be able to check; an alert is for things that need someone to act.

What I'd set up for my own projects

For the API I have running on Container Apps, three alerts would cover the realistic failures: error rate climbing, response time climbing, and the app failing its health check.

That's it. Not CPU, not memory, not the dozen other metrics available, because I wouldn't do anything differently based on them. If CPU is high and nothing is slow and nothing is failing, then nothing is actually wrong.

What I still haven't done

I've studied this rather than run it. Setting up alerts on a live application and then living with them for a while is a different experience from understanding the concepts, and I'd expect that's where I'd learn what my thresholds should actually be. Picking a number for "response time is too high" without real traffic to compare against is a guess, and the only way to fix it is to guess, watch it, and adjust.