The short versionAdding a detector to a noisy environment makes it noisier. The only anomaly product worth having is one that reduces the number of things demanding attention.
Alert fatigue is a correct adaptation
When a team stops reacting to alerts, the usual reading is that the team has gone complacent. The more accurate reading is that the alerts stopped carrying information, and that ignoring a channel which is usually wrong is good judgement rather than poor discipline.
Which means a new detector added to that environment does not start from zero. It starts from below zero, because it is one more source in a channel that has already been discredited. It has to arrive with priority, probable cause and a next action, or it gets filtered out along with everything else — and it will deserve to be.
A team ignoring your alerts has evaluated them. That is the feedback.
What operator-ready means
Four properties, all of them about what the operator is holding at the moment the notification arrives.
- The alert names the affected zone, asset or workload, rather than the metric that moved.
- Comparable historical events come attached, so “is this normal for us” is answerable without starting a search.
- Recommended actions carry their confidence, their tradeoffs and their approval status.
- Resolution is recorded, which is how the next person to meet this pattern inherits the answer instead of re-deriving it.
The number to hold it to
Not detections found. The measure worth watching is how many items are competing for an operator's attention across a shift, and whether that count went down. A product that finds more anomalies and generates more notifications has succeeded at the demo and failed at the job.
It will still miss things, and it will still occasionally escalate something that did not need escalating. The claim is a shorter queue of better-described work — not omniscience.
Written about DataCenterAI
