Thresholds
In this section you will learn how to grade each run of a test from its duration, with thresholds set by hand or computed from past runs.
How thresholds grade a run
A test that passes can still be too slow. Thresholds compare the duration of each run with four limits, in seconds:
| Duration | Threshold status |
|---|---|
| Below Minor | OK |
| Minor or more | MINOR |
| Major or more | MAJOR |
| Critical or more | CRITICAL |
| Timeout or more | TIMEOUT |
Each limit must be greater than the previous one: Minor < Major < Critical < Timeout.
Timeout is also a hard limit: a run still going when it is reached is stopped and recorded as errored. Set it well above the slowest duration you accept. The threshold status is recorded with every run in the performance history and can trigger notifications.
Set thresholds by hand
- Hover over the test in the Tests tab and click Test Settings.
- In the Thresholds tab, turn on Enable Threshold.
- Enter the Minor, Major, Critical and Timeout values in seconds. The Threshold Timeline bar shows the four ranges as you type.
- Click Save Settings.
The default values are 160, 190, 210 and 240 seconds: lower them to match your journey.
Auto-thresholds
Auto-thresholds compute the limits from the test’s own history, so they follow the real performance of the site. They are not available with a free license.
In the Auto-Thresholds tab of Test Settings:
- Turn on Enable Auto-Threshold Computation.
- Set the Analysis Period: the number of days of history to analyze (30 by default).
- Adjust the Threshold Factors if needed.
- Click Save Settings.
Each limit is the average duration of the successful runs of the period plus a number of standard deviations (σ):
| Limit | Default factor |
|---|---|
| Minor | average + 4 σ |
| Major | average + 6 σ |
| Critical | average + 8 σ |
| Timeout | average + 10 σ |
Lower factors make the grading stricter; higher factors tolerate more variation. Enabled auto-thresholds are recomputed every day at midnight, while Pulse is running.
The Manual Computation button computes and applies the thresholds immediately, and shows the sample size, average and standard deviation it used. At least 5 successful runs in the period are needed.
See also
- Notifications — be alerted when a threshold is exceeded
- Performance history — durations and statuses over time
- Transactions — thresholds per stage of a journey in Mugnsoft