Did the defect rate fall after a supplier change?

The audit question

Quality data showed the decrease in average defect rate after the company switched to a new supplier. The quality team is confident that the change is working. Before including this finding / evidence in the audit report, the auditor needs to know: is the improvement real, or could it have occurred by chance?

Scenario

A manufacturing company switched its primary component supplier midway through the year. The quality team reported that defect rate dropped from an average 0.24 to 0.20 in the 15 batches following the switch.

Management celebrated the result and attributed it to to the new supplier. The auditor's task is to assess the control effectiveness - specifically, whether the supplier change is the credible cause of the improvement, or whether the data is too variable to draw that conclusion.

Data

Days (data0)
0.18
0.36
0.15
0.27
0.22
0.41
0.16
0.28
0.12
0.25
0.19
0.24
0.38
0.15
0.28

Days (data1)
0.17
0.35
0.01
0.25
0.22
0.01
0.16
0.28
0.12
0.25
0.19
0.24
0.38
0.15
0.28

What the tool returned

To check whether that improvement holds up, the auditor gathers fifteen defect rates for the period before the new supplier was introduced and fifteen after. Then runs both sets through the tool. The following result came back:

Analyser

🖊️

Before the change (data0), the metric averaged 0.24, typically ranging between 0.1 and 0.4. Since the change (data1), it now averages 0.20. With 94% confidence the true figure sits between 0.1 and 0.3. Results are mixed - 78.1% of samples point to a decrease and 6.5% point to an increase. 15.5% of effects are within negligible-effect zone. The evidence is not yet good enough to draw a firm conclusion.

Size of the change

-0.038

Verdict

Inconclusive

Model: Normal (auto-detected) n₀ = 15 · n₁ = 15 · ESS = 429

Audit conclusion

Defect rate dropped from 0.24 to 0.20 after the supplier switch, and management already credits the new supplier for it. But checking the fifteen batches before aginst the fifteen after shows the data is too noisy to back that up.

So this goes into the report as inconclusive, not as a confirmed win. It shouldn't be used as evidence that the supplier change fixed the defect problem. The recommendation is to keep collecting batch data and reverify the change.

Tool usage benefits

It is easy to accept the evidence without the verification - a drop in defect rate feels like the change of the supplier is working. ChangeVerifier stops that from turning into a true evidence too early. It shows the real split (78.1% vs 6.5% vs 15.5%), flags how much of the result could just be a noise, and calls it inconclusive instead of letting a good-looking average slide through.